show_images([extracted_images[60],extracted_images[80]], None, 10)
What I want
For this purpose,
The extracted images will be processed using cv2. Tesseract is used only for OCR.
import re
import cv2
import math
import numpy as np
import fitz
import pytesseract
import matplotlib.pyplot as plt
%matplotlib inline
Use these functions to display small images within cells.
Using matplotlib to facilitate resizing.
def show_image(img, size=6):
"""
For display single image
"""
plt.figure(figsize = (size, size))
plt.axis('off')
plt.imshow(img)
def show_images(images, titles=None, size=4, spacing=0.03):
"""
For display multiple image in horizontal
"""
num_images = len(images)
if titles == None:
titles = [f'image {i+1}' for i in range(num_images)]
fig = plt.figure(figsize=(size * num_images, size*2))
fig.patch.set_facecolor('lightgray')
for i, (image, var_name) in enumerate(zip(images, titles)):
ax = fig.add_subplot(1, num_images, i + 1)
ax.imshow(image)
ax.axis('off')
ax.set_title(var_name)
ax.xaxis.set_ticks([])
ax.yaxis.set_ticks([])
ax.set_aspect('equal')
fig.subplots_adjust(wspace=spacing)
plt.show()
Images are extracted by byte object, convert to array.
Create image list, use fitz, pyMuPdf
If we just want to extract the image,
cv2.imwrite("path/file_name.png", image_file)
finish with this.
Continue to clean the image and turn it into text.
def image_extract(pdf_file):
pdf = fitz.open(pdf_file)
num_of_pics = 0
extract_list = []
for i, page in enumerate(pdf):
images = page.get_images()
rotate = page.rotation
if not len(images) == 0:
for image in images:
num_of_pics += 1
xref = image[0]
img = pdf.extract_image(xref)['image']
nparr = np.frombuffer(img, np.uint8)
cv_image = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
if rotate == 90:
cv_image = cv2.rotate(cv_image, cv2.ROTATE_90_CLOCKWISE)
elif rotate == 270:
cv_image = cv2.rotate(cv_image, cv2.ROTATE_90_COUNTERCLOCKWISE)
extract_list.append(cv_image)
print("========= Done ===========")
print(f"Total of {num_of_pics} images from {pdf_file}")
return extract_list
%%time
input_file = 'mm_gbfl.pdf'
extracted_images = image_extract(input_file)
========= Done =========== Total of 123 images from mm_gbfl.pdf CPU times: user 32.9 s, sys: 2.86 s, total: 35.7 s Wall time: 36.5 s
show_images([extracted_images[60],extracted_images[80]], None, 10)
In the extracted image, the left and right pages were combined into one image.
The left page appears to be skewed to the left, and the right page appears to be skewed to the right.
Since the left and right sides are at different angles, each page will be separated.
Cut the image by half width.
def cut_off(cv_image):
h, w, _ = cv_image.shape
cutoff = w // 2
left = cv_image[:, :cutoff]
right = cv_image[:, cutoff:]
return left, right
%%time
cutoff_images =[cut_off(image) for image in extracted_images]
len(cutoff_images[0])
CPU times: user 334 µs, sys: 827 µs, total: 1.16 ms Wall time: 1.19 ms
2
title = ['Page Left', 'Page Right']
show_images(cutoff_images[60], title)
Split the image into left and right parts using a function.
The left and right lists are returned, so this list is nested.
flat_images = [image for images in cutoff_images for image in images]
print(f"Total {len(flat_images)} images")
Total 246 images
Look at the colors most often used to compensate for the space created by rotation: the background color.
Mostly use white, but check the specific color number, such as 255 or 254 or 253 etc.
Only one randomly selected page will be displayed.
def get_bg_col(cv_image):
image = cv_image.copy()
flat_image = cv_image.reshape(-1, 3)
colors, counts = np.unique(flat_image, axis=0, return_counts=True)
bg_color = colors[np.argmax(counts)]
return bg_color
%%time
bg_color = get_bg_col(flat_images[36])
print(f"Most often color is {bg_color}")
Most often color is [255 255 255] CPU times: user 5.73 s, sys: 87.7 ms, total: 5.82 s Wall time: 6.17 s
It found that this image uses white 255 255 255
If image size is large or small, the detection range will be finer, so make the image better to make the detection range rougher.
def scale_image(cv_image, val=500):
height, width, _ = cv_image.shape
if height > val or width > val:
scale = val / max(width, height)
cv_image = cv2.resize(cv_image, None, None, scale, scale)
else:
scale = 1
return cv_image, scale
Detect straight lines using cv2.houghLinesP.
Rotates an image by calculating an angle from a straight line.
There are several ways to rotate, but I will use affine because it is fast.
The downscaled image is used to obtain a threshold value.
After calculating the angle using the threshold, rotate the original image before downscaling.
Since -90, 90 degrees is detected and the angle is incorrect, this case limited the detection angle to 0 to 45 degrees.
def calculate_angle(cv_image, val=500):
image, _ = scale_image(cv_image, val)
img_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
img_edges = cv2.Canny(img_gray, 100, 100, apertureSize=3)
lines = cv2.HoughLinesP(img_edges, 1, math.pi / 180.0, 100, minLineLength=100, maxLineGap=5)
if lines is not None:
angles = []
for [[x1, y1, x2, y2]] in lines:
angle = math.degrees(math.atan2(y2 - y1, x2 - x1))
angles.append(angle)
degree = np.median(angles)
else: degree = 0
return degree
Let's see how well angles such as 90, -90, 0, 45 can be detected as errors.
%%time
for val in range(400, 2400, 200):
degrees = [calculate_angle(img, val) for img in flat_images]
percent = len([deg for deg in degrees if abs(deg) >= 45 or abs(deg) ==0]) / len(degrees) *100
print(f'Scale value {val} : Detect error {percent} %')
Scale value 400 : Detect error 46.34146341463415 % Scale value 600 : Detect error 45.9349593495935 % Scale value 800 : Detect error 45.52845528455284 % Scale value 1000 : Detect error 48.78048780487805 % Scale value 1200 : Detect error 47.5609756097561 % Scale value 1400 : Detect error 43.90243902439025 % Scale value 1600 : Detect error 39.83739837398374 % Scale value 1800 : Detect error 43.08943089430895 % Scale value 2000 : Detect error 50.40650406504065 % Scale value 2200 : Detect error 60.97560975609756 % CPU times: user 3min 6s, sys: 33.7 s, total: 3min 40s Wall time: 3min 19s
The larger the image, the longer the calculation will take.
In the above results, 1600 was the best value.
def fix_angle(cv_image):
degree = calculate_angle(cv_image)
h, w, _ = cv_image.shape
center = (w // 2, h // 2)
col = (255, 255, 255)
if abs(degree) < 45:
rotation_matrix = cv2.getRotationMatrix2D(center, degree, 1.0)
cv_image = cv2.warpAffine(cv_image, rotation_matrix, (w, h), borderValue=col)
return cv_image
%%time
rotate_images = [fix_angle(image) for image in flat_images]
CPU times: user 16.6 s, sys: 6.56 s, total: 23.1 s Wall time: 13.1 s
page = 128
images = [flat_images[page],
rotate_images[page],
flat_images[page+1],
rotate_images[page+1],
flat_images[page+2],
rotate_images[page+2]
]
titles = ['Before','After']*(len(images)//2)
show_images(images, titles)
It can see that the tilt has been corrected whether the tilt is to the right or to the left.
Let's use scipy.ndimage to rotate the image and check the speed.
from scipy import ndimage
def fix_angle_ndimage(cv_image):
degree = calculate_angle(cv_image)
if abs(degree) < 45:
cv_image = ndimage.rotate(cv_image, degree, reshape=False, cval=255)
return cv_image
%%time
rotate_images_ndimage = [fix_angle_ndimage(img) for img in flat_images]
CPU times: user 7min 25s, sys: 11.7 s, total: 7min 37s Wall time: 8min 19s
With warpAffine it took total 13.1 second, but with ndimage it took 8min 19s.
The difference is obvious, it takes 38 times as long with this 246 images.
The same is true when viewing the results.
page = 128
show_images([
flat_images[page],
rotate_images_ndimage[page],
flat_images[page+1],
rotate_images_ndimage[page+1],
flat_images[page+2],
rotate_images_ndimage[page+2],
], titles)
Now that we have corrected the distortion, we can crop the image.
There is usually a mix of different layouts on each page.
However, if it is a book, the print area should basically be the same on all pages.
So, if we can get that range, can crop it.
Create an image by overlapping the corrected images to obtain the range.
Make the background color transparent so that only the printed part is visible.
def white_transparent(cv_image):
h, w, c = cv_image.shape
image_bgra = np.concatenate([cv_image, np.full((h, w, 1), 255, dtype=np.uint8)], axis=-1)
parm = [255, 255, 255]
white = np.all(cv_image == parm, axis=-1)
image_bgra[white, -1] = 0
return image_bgra
To create a single image from a list of images, pass the list directly.
Create a base image from the first page.
Depending on the book, the first and last four or five pages may have almost no text or be blank. So, in this case randomly select images and overlay them.
import random
def stack_images(images, pages=10):
base = white_transparent(images[0])
shuffled_images = random.sample(images, len(images))
for image in shuffled_images[:pages]:
img = white_transparent(image)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)[1]
masked = cv2.bitwise_and(img, img, mask=255-thresh)
bitwised = cv2.bitwise_and(base, base, mask=thresh)
base = cv2.add(bitwised, masked)
return base
%%time
random_stacked_image = stack_images(rotate_images, 20)
CPU times: user 4.2 s, sys: 576 ms, total: 4.77 s Wall time: 4.91 s
show_image(random_stacked_image)
The processing method
Reduce the size of the image if it is large. Set the maximum pixel value as the value of val.
Create the size to detect with detect_area.
$ width \times height \times scale \times scale = \text{Area of Original Image} $, which is then multiplied by the number filter_size required for the filter.
This time it is 0.5, so if it is 50% area, it will be considered a box.
Adjust the block_size, and c values to change threshold to to polarize the image.
Use findContours to detect contours.
From the detected contour, select only the values that have an area larger than detect_area for contourArea.
Get the coordinates with boundingRect. This values are the coordinate of the scaled image, so it must be rescaled to accommodate the original image.
Add the rescaled box values to the list using the internally defined function scaler.
To see image, use rectangle and show_image func.
def detect_frame(cv_image, val=600):
cv_image = cv_image[:,:,:3].copy()
cv_image, scale = scale_image(cv_image, val)
height, width, _ = cv_image.shape
c = 9 # Constant subtracted from calculated threshold
block_size = 47 # Size of blocks to split the image, must be an odd number
filter_size = 0.5 # Filter small area less than filter_size of image
detect_area = width * height * scale * scale * filter_size
img_gray = cv2.cvtColor(cv_image, cv2.COLOR_BGR2GRAY)
img_thre = cv2.adaptiveThreshold(img_gray, 255,
cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY_INV,
block_size, c)
contours, _ = cv2.findContours(img_thre, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
def scaler(value):
return int(value/scale)
box = []
for i, contour in enumerate(contours):
if cv2.contourArea(contour) > detect_area:
x, y, w, h = cv2.boundingRect(contour)
box.append((scaler(x), scaler(y), scaler(w), scaler(h)))
cv2.rectangle(cv_image,(x,y),(x+w,y+h), (255,0,0), 2)
show_image(cv_image)
return box
%%time
boxes = detect_frame(random_stacked_image)
boxes
CPU times: user 59.5 ms, sys: 16.2 ms, total: 75.7 ms Wall time: 86.8 ms
[(164, 156, 1307, 2097)]
I got the box just fine. Then use this box to crop the image.
Crop the image by specifying the value in the box.
If the edge of the screen and the text are the same, it will be difficult to read, so 1.25% of the width of the image is calculated as the margin.
This is based on the information that a typical book has about 80 characters per line, and this is to provide a margin of one character.
def crop_image(cv_image, box):
height, width, _ = cv_image.shape
margin = width//80 # margin 1.25 percent of width
x1, y1, x2, y2 = box[0]
x = x1 - margin
y = y1 - margin
w = x2 + x1 + margin
h = y2 + y1 + margin
cropped_image = cv_image[y:h, x:w]
return cropped_image
%%time
cropped_images = [crop_image(image, boxes) for image in rotate_images]
CPU times: user 272 µs, sys: 10 µs, total: 282 µs Wall time: 286 µs
page = random.randint(0, len(cropped_images) - 1)
titles = ['Extracted Image','Rotated Image','Cropped Image']
show_images([
flat_images[page],
rotate_images[page],
cropped_images[page]
],titles)
Pretty good. Result margins, left and right are depends on original layout of pages.
Now that we have created a clean image with the distortion corrected, we will use this data to extract the text by Tesseract.
Adjust parameters in config. (https://tesseract-ocr.github.io/tessdoc/ImproveQuality.html)
def ocr_text(image):
config = r'--oem 3 --psm 3'
ocr_text = pytesseract.image_to_string(image, lang='eng', config=config)
return ocr_text
%%time
extracted_texts = [ocr_text(image) for image in cropped_images]
CPU times: user 1min 33s, sys: 6.22 s, total: 1min 39s Wall time: 8min 46s
Since there are 246 pages and 8m46s, each OCR page takes 2.13s.
Text tidy by replacing hyphens in word wrap, line breaks before periods, and more.
replacements:
def tidy_text(text):
replacements = {
r'([\w])-\n(\w)': r'\1\2',
r'(—)\n+(\w)': r'\1\2',
r'([\w\;\:\,\)\]\”\’])\n([\(\{\“\‘\w])': r'\1 \2',
}
for key in replacements.keys():
text = re.sub(key, replacements[key], text)
return text
%%time
jointed_text = ''.join(extracted_texts)
result_text = tidy_text(jointed_text)
CPU times: user 46.7 ms, sys: 2.82 ms, total: 49.6 ms Wall time: 50.9 ms
print('Length of Extracted Text List :', len(extracted_texts))
print('Length of Text file Character :', len(result_text))
Length of Extracted Text List : 246 Length of Text file Character : 442820
print(result_text[9000:12000])
worthy sentiments, but worse than useless if one’s actions do not support them. As opposed to those leaders who use language only as a disguise, the people on whose experience this book is based actually have shown that they tried hard to create ethical environments in which individuals could realize their potentials. By no means did they always live up to their stated intentions; but their ideas, words, and example show that doing business can be much more fulfilling than most of us realize. So, based in large part on the experience of these exemplary CEOs, this book will discuss what it means to be a good leader, a good manager, and a good worker. Bookstores are full of volumes containing very good advice about how to be an effective manager or successful leader. Often such *See note 1, p. 213. books will instruct a reader to model his or her behavior on the cynical wisdom of Machiavelli, the relentless drive of Genghis Khan, or the ruthlessness of Attila the Hun as a way to achieve power and plunder. Good Business has a more modest ambition. It will explore how leaders who have impressed their peers for both their business success and their commitment to broader social goals go about their jobs—what ambitions motivate them, and what kind of organizations they try to develop in pursuit of those ideals. The necessity for considering such issues is simple: Today business leaders are among the most influential members of society. While they are all trained to generate profits, many of them are oblivious to the other responsibilities that their new societal leadership entails. In this book, visionary leaders will explain what they consider to be their duties and how they go about fulfilling them. In the process of examining their philosophies and their practical applications, we will focus especially on how leaders and managers and even the concerned employees of any organization can learn to contribute to the sum of human happiness, to the development of an enjoyable life that provides meaning, and to a society that is just and evolving. These may seem like goals that are beyond human reach, and certainly outside the scope of a book dealing with business. But the way we make a living, the jobs we have, and the way our work is rewarded have a tremendous bearing on our lives, making them either exciting and rewarding, or dull and anxious. For that reason alone anyone in charge of a workplace is obliged to consider the question: How am I contributing to human well-being? This is certainly not a concern that motivated Genghis Khan, or even Machiavelli. But to follow the examples of such social predators prevents business leaders from achieving their full potential. Of course there will always be single-mindedly ambitious executives concerned only with clawing their way to the top. But is such behavior really the kind of leadership we want in our society? In fact, there are enough people in business who do genuinely value organizations that promo
Although it's not perfect, I was able to output neat text for each paragraph.
If annotations like a *See note 1, p. 213. in above output was inserted in the middle or bottom of page, connection between sentences become unclear.
Then that text data can output to a file.
Defines the name of the output file.
Here, I will delete extension .pdf from input file name and use it.
Add ocr_text_from_ in front of the obtained name, at last add .txt .
original_file_name = f'{input_file}'[:-4]
output_file_name = 'ocr_text_from_' + original_file_name + '.txt'
with open(output_file_name, "w", encoding="utf-8") as f:
f.write(result_text)
print(f"File '{output_file_name}' was SAVED")
File 'ocr_text_from_mm_gbfl.txt' was SAVED
The time taken by the function used to extract the text
| Function | Time (Second) | Note |
|---|---|---|
| image_extract | 36.5 | |
| cut_off | 0.01 | option |
| fix_angle | 13.1 | warpAffine |
| get_bg_col | 6.17 | |
| stack_images | 4.91 | |
| detect_frame | 0.08 | |
| crop_image | 0.00 | |
| ocr_text | 526 | 8min46s |
| tidy_text | 0.05 |
The background color is generally white, so if you specify the basic color as 255, you can omit get_bg_col.
I think it is also possible to obtain frames from one image by adjusting parameters and expanding the image without using stack_image.
Tesseract OCR takes 2 seconds per page, and if you have a small number of pages you won't have to wait that long.