Blog

Blog

Suppose, you need to go through a dataset to pick images for your classification training data.

cv-cat view can be used for basic browsing, selecting images, and assigning a numeric label to them:

cat images.bin | cv-cat "view=0,,png"

The command above will show the image and wait a key press:

Press whitespace to save the file as <timestamp>.png, e.g. 20170101T123456.222222.png

Press numerical keys 0-9: save the file as <timestamp>.<num>.png, e.g. if you press 5: 20170101T123456.222222.5.png

Press <Esc> to exit.

Press any other key to show the next frame without saving.

 

view parameters have the following meaning:

The first parameter is the wait in milliseconds for key press, 0 to wait indefinitely.

The second parameter is the window title (irrelevant for labelling).

The third parameter is the image extension e.g. png, jpg ...; default ppm.

 

A few new features have been added to cv-cat accumulate filter.

Before, it accumulated images as sliding window of a given size. Now, you could also ask for fixed layout of the accumulated image. It sounds confusing, but try to run the following commands (press any key to move to the next image)

> # make sense of the input
> ( yes 255 | csv-to-bin ub ) | cv-cat --input 'no-header;rows=64;cols=64;type=ub' 'count;view=0;null'
> # accumulate as sliding window of size 4
> ( yes 255 | csv-to-bin ub ) | cv-cat --input 'no-header;rows=64;cols=64;type=ub' 'count;accumulate=4;view=0;null'
> # accumulate as sliding window of size 4 in reverse order
> ( yes 255 | csv-to-bin ub ) | cv-cat --input 'no-header;rows=64;cols=64;type=ub' 'count;accumulate=4,,reverse;view=0;null'
> # accumulate images in fixed order
> ( yes 255 | csv-to-bin ub ) | cv-cat --input 'no-header;rows=64;cols=64;type=ub' 'count;accumulate=4,fixed;view=0;null'
> # accumulate images in fixed order (reverse)
> ( yes 255 | csv-to-bin ub ) | cv-cat --input 'no-header;rows=64;cols=64;type=ub' 'count;accumulate=4,fixed,reverse;view=0;null'

For example, if you want to create an image from fixed number of tiles, you could run something like this:

> ( yes 255 | csv-to-bin ub ) | cv-cat --fps 1 --input 'no-header;rows=64;cols=64;type=ub' 'count;accumulate=4,fixed;untile=2,2;view=0;null'

Say, you process images, but would like to view them in the middle of your pipeline in a different way (e.g. increase their brightness, resize, etc).

Now, you can do it with cv-cat tee. For example:

> # make a test image
> ( echo 20170101T000000,64,64,0 | csv-to-bin t,3ui; yes 255 | head -n $(( 64 * 64 )) | csv-to-bin ub ) > image.bin
> # observe that the images viewed in tee are passed unmodified down the main pipeline for further processing
> for i in {1..100}; do cat image.bin; done | cv-cat --fps 1 "count;tee=invert|view;resize=2" | cv-cat "view;null"

You could specify (almost) any pipeline in your tee filter, but viewing and, perhaps, saving intermediate images in files seem so far the main use cases.

Recently, we found that cv-cat view stopped working properly, when used several times in the same cv-cat call.

Something like

> cat images.bin | cv-cat "view;invert;view;null"

would either crash or behave in undefined way. All our debugging has pointed to some sort of race condition in the underlying cv::imshow() call or deeper in X-windows-related stuff, thus, at the moment, it seems to be out of our control.

Use the following instead:

> cat images.bin | cv-cat "view;invert" | cv-cat "view;null"

cv-cat is now able to perform pixel clustering by color using the k-means algorithm.

for example:

> cv-cat --file rippa.png "convert-to=f,0.0039;kmeans=4;view;null" --stay

input image:

output image (4 clusters):

A new convenience utility ros-from-csv is now available in snark. It reads CSV records and converts them into ROS messages with the usual conveniences of csv streams (customised fields, binary format, stream buffering/flushing, etc).

Disclaimer: ros-from-csv is a python application and therefore may not perform well streams that require high bandwidth or low latency.

You could try it out, using the ROS tutorial Understanding Topics (http://wiki.ros.org/ROS/Tutorials/UnderstandingTopics):

Run ROS Tutorial nodes:

> # in a new shell
> roscore
> # in a new shell
> rosrun turtlesim turtle_teleop_key

 

Send your own messages on the topic, using ros-from-csv:

> echo 1,2,3,4,5,6 | ros-from-csv /turtle1/cmd_vel

Or do a dry run:

> echo 1,2,3,4,5,6 | ros-from-csv /turtle1/cmd_vel --dry
linear: 
  x: 1.0
  y: 2.0
  z: 3.0
angular: 
  x: 4.0
  y: 5.0
  z: 6.0

You also can explicitly specify message type:

> # dry run
> echo 1,2,3 | ros-from-csv --type geometry_msgs.msg.Point --dry
x: 1.0
y: 2.0
z: 3.0
 
> # send to a topic
> echo 1,2,3 | ros-from-csv --type geometry_msgs.msg.Point some-topic

A new convenience utility ros-to-csv is now available in snark. It allows to output as CSV the ROS messages from rosbags or from topics published online.

You could try it out, using the ROS tutorial Understanding Topics (http://wiki.ros.org/ROS/Tutorials/UnderstandingTopics):

Run ROS Tutorial nodes:

> # in a new shell
> roscore
> # in a new shell
> rosrun turtlesim turtlesim_node
> # in a new shell
> rosrun turtlesim turtle_teleop_key

Run ros-to-csv; then In the shell where you run turtle_teleop_key, press arrow keys to observe something like:

> # in a new shell
> ros-to-csv /turtle1/cmd_vel --verbose
ros-to-csv: listening to topic '/turtle1/cmd_vel'...
0,0,0,0,0,-2
0,0,0,0,0,2
-2,0,0,0,0,0
-2,0,0,0,0,0
0,0,0,0,0,2
0,0,0,0,0,-2
2,0,0,0,0,0
0,0,0,0,0,2

If you log some data in a rosbag:

> # in a new shell
> rosbag record /turtle1/cmd_vel

You could convert it to csv with a command like:

> ros-to-csv /turtle1/cmd_vel --bag 2017-11-06-14-43-34.bag
2,0,0,0,0,0
0,0,0,0,0,-2
-2,0,0,0,0,0
0,0,0,0,0,2
0,0,0,0,0,2
0,0,0,0,0,-2
2,0,0,0,0,0

Sometimes, you have a large file or input stream that is mostly sorted, which you would like to fully sort (e.g. in ascending order).

More formally, suppose, you know that for any record Rn in your stream and any records Rm such that m - n > N, Rn < Rm, where N is constant.

Now, you can sort such a stream, using csv-sort, --sliding-window=<N>:

 

> ( echo 3echo 1; echo 2; echo 5echo 4 ) | csv-sort --sliding-window 3 --fields a
0
1
2
3
> ( echo 4echo 5echo 2echo 1echo 3 ) | csv-sort --sliding-window 3 --fields a --reverse
3
2
1
0

As usual, you can sort by multiple key fields (e.g. csv-sort --sliding-window=10 --fields=a,b,c), sort block by block (e.g. csv-sort --sliding-window=10 --fields=t,block), etc.

Sometimes, you have a large file or input stream that is mostly sorted by some fields with just a few records out of order now and then. You may not care about those few outliers, all you want is most of your data sorted.

Now, you can discard the records out of order, using csv-sort, e.g:

> ( echo 0; echo 1; echo 2; echo 1; echo 3 ) | csv-sort --discard-out-of-order --fields a
0
1
2
3
> ( echo 3; echo 2; echo 1; echo 2; echo 0 ) | csv-sort --discard-out-of-order --fields a --reverse
3
2
1
0

As usual, you can sort by multiple key fields (e.g. csv-sort --discard-out-of-order --fields=a,b,c), sort block by block (e.g. csv-sort --discard-out-of-order --fields=t,block), etc.

The ratio and linear-combination operations of cv-cat have been extended to support assignment to multiple channels. Previously, these operations would take up to 4 input channels (symbolically always named r, g, b, and a, regardless of the actual contents of the data) and produce a single-channel, grey-scale output. Now you can assign up to four channels:

ratio syntax
... | cv-cat "ratio=(r-b)/(r+b),(r-g)/(r+g),r+b,r+g"

The right-hand side of the ratio / linear combination operations contains comma-separated expressions defining each of the output channels through the input channels. The number of output channels is the number of comma-separated fields, it may differ from the number of input channels. As a shortcut, an empty field, such as in

ratio syntax shortcut
... | cv-cat "ratio=,r+g+b,"

is interpreted as channel pass-through. In the example above the output has three channels, with channels 0 and 2 assigned verbatim to the input channels 0 and 2 (r and b, symbolically), and the channel 1 (symbolic g) assigned to the sum of all three channels.

As yet another shortcut, cv-cat provides a shuffle operation that re-arranges the input channels without changing their values:

shuffle syntax
... | cv-cat "shuffle=b,g,r,r"

In this case, the order of the first 3 channels is reversed, while the former channel r is also duplicated into channel 3 (alpha). Internally, shuffling is implemented as a restricted case of linear combination, and therefore, other usual rules apply: the number of output channels is up to 4, it does not depend on the number of input channels, and an empty field in the right-hand side is interpreted as channel pass-through.

When using view-points, there often is a need to quickly visualise or hide several point clouds or other graphic primitives.

Now, you can group data in view-points, using groups key word. A source can be assigned to one or more groups by using the groups arguments. Basic usage is:

view-points "...;groups=g1,g2"

For example if we have two graphs as follows:

$ cat <<EOF > edges01.csv
1,1,0,4,4,0
4,4,0,4,8,0
4,4,0,8,4,0
EOF

$ cat <<EOF > nodes01.csv
1,1,0,Node00
4,4,0,Node01
4,8,0,Node02
8,4,0,Node03
EOF

$ cat <<EOF > edges02.csv
4,9,1,4,12,1
4,12,1,0,9,1
4,9,1,0,4,1
0,4,1,0,9,1
EOF

$ cat <<EOF > nodes02.csv
0,4,1,Node20
0,9,1,Node21
4,9,1,Node22
4,12,1,Node23
EOF

We can separate the graphs as well as group together nodes and edges of different graphs as follows:

$ view-points "nodes01.csv;fields=x,y,z,label;colour=yellow;weight=5;groups=graph01,nodes,all" \
	"edges01.csv;fields=first/x,first/y,first/z,second/x,second/y,second/z;shape=line;colour=yellow;shape=line;groups=graph01,edges,all" \
	"nodes02.csv;fields=x,y,z,label;colour=green;weight=5;groups=graph02,nodes,all" \
	"edges02.csv;fields=first/x,first/y,first/z,second/x,second/y,second/z;shape=line;colour=green;shape=line;groups=graph02,edges,all"

Try to switch on/off checkboxes for various groups (e.g. "graph01", "nodes", etc) and observe the effect.

A quick note on new operations in cv-calc utility. Time does not permit to present proper examples, but hopefully, cv-calc --help would be sufficient to give you an idea.

cv-calc grep

Output only those input images that conform a certain condition. Currently, only min/max number or ratio of non-zero pixels is supported, but the condition can be any set of filters applied to the input image (see cv-cat --help --verbose for the list of the filters available).

Example: Output only images that have at least 60% of pixels darker than a given threshold:

> cat images.bin | cv-calc grep --filters="convert-to=f,0.0039;invert;threshold=0.555" --non-zero=ratio,0.6

cv-calc stride

Stride to the input image with a given kernel (just like a convolution stride), output resulting images.

cv-calc thin

Thin the image stream by a given rate or a desired frames-per-second number.

csv-shape is a new utility for various operations on reshaping csv data.

For now, only one operation is implemented: concatenate:

Concatenate by Grouping Input Records

> ( echo 1,a; echo 2,b; echo 3,c; echo 4,d; ) | csv-shape concatenate -n 2
1,a,2,b
3,c,4,d

Note: For ascii text inputs the records do not have to be regular or even have the same number of fields.

Concatenate by Sliding Window

ASCII:

> ( echo 1,a; echo 2,b; echo 3,c; echo 4,d; ) | csv-shape concatenate -n 2 --sliding-window
1,a,2,b
2,b,3,c
3,c,4,d

Binary:

> ( echo 1,a; echo 2,b; echo 3,c; echo 4,d; ) | csv-to-bin ui,c | csv-shape concatenate -n 2 --sliding-window --binary ui,c | csv-from-bin ui,c,ui,c
1,a,2,b
2,b,3,c
3,c,4,d

This is a brief introduction to cv-cat new filters:

 

Filter: Accumulated

This filter is used to calculate pixel-wise (and channel-wise) average from the sequential series of input images.

As it relies on the sequential accumulated input images, this filter is run in serial mode in cv-cat. This as implications when used with 'forked' image processing.

However parallel processing is utilised on image rows dimension.

Please download the following file which contains a total of 8 images: images.bin: 8 images showing movement. Viewing the images:

cat images.bin | cv-cat "view=250;null" 


Average:

Calculating averages using all accumulated input images, the output is also 8 images.

cat images.bin | cv-cat "accumulated=average;view=250;null"

The 6th output image is the average of all 6 accumulated images, the 7th is the average of the 7 accumulated input images.

 

Exponential Moving Average (EMA):

Calculating the average using a sliding window of images. Here a sliding window of 3 images is used.

cat images.bin | cv-cat "accumulated=average,3;view=250;null"

The output is 8 images, the 6th image is the accumulation of image 1 to 6. Please research the simple EMA formula.

 

Forked Arithmetic Filters: Multiply, Divide, Add and Subtract

This group of filters work similar to the mask filer: Masking images with cv-cat, they both use a sub-filters to generate a mask or operand image. 

A mask has values of 0 or '> 0' mask file to apply to the image, a corresponding pixel in the mask with a value of 0 is masked. The arithmetic filters work on operand images where is pixel value is important.

Multiply:

This filter will do pixel-wise multiplication the operand image and the input image. It wraps cv::multiply function.

Please download this simple mask file: scaled-3f.bin

#Viewing the mask
cat scaled-3f.bin | cv-calc header
cat scaled-3f.bin| cv-cat "view=1000;null"

Applying a single scaled image to the input images:

cat images.bin | cv-cat "multiply=load:scaled-3f.bin;view=250;null"

You should see images similar to below. scaled-3f.bin has values in the range of 0 to 1.0, the command above will darken the images.

From the example above: cv-cat's multiply is run in parallel, multiple input images are applied the scaled-3f.bin file in parallel.

This is because the all sub-filter(s) can run in parallel mode, in this case there is only one sub-filter 'load'.

The example below also shows multiply running in parallel mode as as load and threshold are parallel-able filters.

cat images.bin | cv-cat "multiply=load:scaled-3f.bin|threshold:0.7,1;view=250;null"


Subtract:

This filter simply subtract the operand image from each input image. This is a wrapper to cv::subtract.

The operand image is derived from the sub-filters. In this example we shall use the accumulated filter mentioned earlier. This is a simple method for detecting moving objects in the image.

cat images.bin| cv-cat "subtract=accumulated:average,3;brightness=5;view=250;null"

Each input image is subtracted the EMA average, where the EMA window is 3.

You should see similar images shown below:

In the example above: the multiply filter is run in serial mode. This is because one of the sub-filter or sub-filters ('accumulated' in this case) can only be run in serial mode.

If you have a webcam handy or it is built into the laptop, try this command:

cv-cat --camera "subtract=accumulated:average,10;view;null"



Add:

This is a wrapper to cv::add

This filter is the opposite of subtract. In this case if you add the EMA average (the "background") to the input images. Any moving object becomes transparent.

cat images.bin| cv-cat "add=accumulated:average,3;view=250;null"

This is the result:

Of course you can always try this pipeline with a physical camera:

cv-cat --camera "subtract=accumulated:average,10;view;null"

 

Divide:

This filter wraps cv::divide, divides the input images by the operand.

The file scaled-3f.bin has values in the range 0 to 1.0, hence dividing the image by scaled-3f.bin will brighten the image.

Arithmetic filters: the output image type is the same as the input image type.

cat images.bin| cv-cat "divide=load:scaled-3f.bin;view=250;null"

 

 

A brief notification on the latest additions to cv-cat (and all other camera applications linking in the same filters).

As of today, the application provides access to all the morphology operations available in OpenCV:

  1. erosion
  2. dilation
  3. opening
  4. closing
  5. morphological gradient
  6. top-hat
  7. black-hat

See OpenCV documentation for more details. In addition, a skeleton (a.k.a. thinning) filter is implemented on top of the basic morphological operations. The implementation follows this demo. However, this is neither the fastest nor the best implementation of thinning. Possibly the optimal approach is proposed in the paper "A fast parallel algorithm for thinning digital patterns" by T.Y. Zhang and C.Y. Suen. See this demo for comparative evaluation of several thinning algorithms (highly recommended!)

Some examples of usage are given below.

Erosion

Input image

Processing

erosion
cv-cat --file spots.png "erode=circle,9,;encode=png" --output=no-header > eroded.png

Result

Multiple Iterations

OpenCV allows multiple iterations of the same morphology operation, the default iterations number is 1. Below is the same erosion operation applied twice (please see cv-cat's help):

cv-cat --file spots.png "erode=circle,9,,2;encode=png" --output=no-header > eroded-twice.png

Result

Thinning

Input image

Processing

thinning
cv-cat --file opencv-1024x341.png "channels-to-cols;cols-to-channels=0,repeat:3;skeleton=circle,3,;encode=png" --output=no-header > skeleton.png

Result