How to avoid overflow in embedded python block in GNURadio?

Question

I am currently working on project which utilizes RTL-SDR for capturing packets. I want to capture packets incoming on two different channels. I created the following RF path and it works just fine in terms of capturing packets and converting them into digital form.

However I want to scrap the payloads of those packets. I correlate the SFD and the Correlate Access Code block outputs in strings of zeros and ones, number two whenever SFD is matched. Knowing that I wrote a custom python block with simple if statement which is looking for those number two's. And this is where problem begins. Without my custom packet parser block everything works just fine, but with it in RF path, flowgraph immediately gets overloaded and barely any packets are captured. I have placed the implementation of packet parse block below. Why is this block causing such massive overload? It's a really simple implementation of a block. I know that samle rate is a troublemaker when we deal with overloading but I am not really willing with going down with sample rate any further. Is there a way of allowing GNURadio to use more CPU? GNURadio is using only 13% of my CPU. I am glad to give more however I don't know how. I was also thinking about usage of multithreading - running multiple concurrent loops which would partition and scrap incoming collections of bits.

class blk(gr.sync_block):
def __init__(self, path='somePath', channel='channel'):  

    gr.sync_block.__init__(
        self,
        name='Packet parser',   
        in_sig=[np.byte],
        out_sig=[]
    )

    self.path = path
    self.channel = channel

def work(self, input_items, output_items):

    for i in range(len(input_items[0])):

        if input_items[0][i] == 2:  
            conn = sqlite3.connect("anotherPath")
            timestamp = str(datetime.datetime.now())
            input_items[0][i] = 0
            somePayloadBitsChanged = self.littleEndianToBig(input_items[0][i:i+64])
            conn.execute(f"INSERT INTO Log (Guid,Timestamp,Channel,SomePayloadBitsChanged ) VALUES ('{uuid.uuid4()}', '{timestamp}', '{self.channel}', '{somePayloadBitsChanged } )");
            conn.commit()
            conn.close()

    return 0

def littleEndianToBig(self, somePayloadBits):

    try:
        array = np.array(somePayloadBits)
        array = np.packbits(array, bitorder='big')

        somePayloadBitsChanged = array[0]
    except:
        print("littleEndianToBig error")
        return 0

    return somePayloadBitsChanged 

Answer 1

1. Don't open and close the database connection for every insert; open it in __init__ (or better, a start method — thanks Marcus Müller), and close it in a stop method.

2. Don't commit once per row inserted; once per work call should be more than enough. Batching up values until you have, say, a thousand, and flushing them all in a single statement would probably be even better.

3. Consider using pragma synchronous=OFF so that sqlite doesn't block gnuradio on fsync.

4. There's almost certainly a more efficient way to write your littleEndianToBig function... especially since it consumes 64 bits and you're only returning 8 of them. At the very least you could save numpy some work by only giving it the 8 bits you're going to use, but there may be a better way (also, I think the function name is misleading).

5. If that's not enough, you could do all of the sqlite work in a separate worker thread that receives values from the block using a thread queue.

6. Or just write the relevant data out to a file in the rawest practical form and postprocess it later, which will probably be a thousand times faster than all this.

Answer 2

You're making one mistake that totally blocks your whole processing and means it can't work, and as explained in Hobbs' answer, which misses that mistake, commit a lot of performance sins.

The Blocker

If your work returns 0 in any case, your block never consumes any input and your flowgraph immediately stalls. If you're not aware of that, maybe it's time to revisit the official GNU Radio tutorials, and check what the work function does, specifically.

The Bug that Indicates Confusion

Also, I don't know what the intention behind

            input_items[0][i] = 0

is, but it's utterly illegal: you must never modify the input to a block! It also makes no sense, because you're not going to read it again if you constructed your python block correctly.

The Performance Sins

Other than that, as hobbs says, opening and closing a database in the work function is a terrible idea. The work function is called thousands and thousands of times during the lifetime of your flowgraph, and you win absolutely nothing by reopening every time. So, in __init__, do something like self.conn = sqlite3.connect("anotherPath"), and use self.conn from thereon. Even better, do it in the def start(self): that you can add to your class, as that is executed shortly before the flowgraph starts running. Close the connection in the def stop(self): method.

Your UUID entry makes little sense here. Your database has functionality to insert an unambigous key for every row, automatically. Use that instead!

You can of course use the stringified timestamp, but I found it usually way more useful (and faster, too!) to use the clock_gettime nanosecond clock, and store these as 64-bit integer. Convert to the readable clock format on retrieval! Doing it that way enabled easy sorting, stores way less data, and doesn't have any of the potential timezone/stringconversion/locale bugs your solution might exhibit. So, instead of timestamp = str(datetime.datetime.now()), use timestamp = time.time_ns() and be done with it. Of course, your Timestamp column needs to get the Int type (instead of Text / Varchar). In a proper relational database system, you'd also avoid storing the string "864.4 MHz" in every row. Instead, you'd have a channels database, check at startup whether that string is already in there, and then just use the ID of that row as Channel value in your Log table.

Generally, avoid putting your values into the SQL string, as that requires numerics->string conversion, followed by string tokenization, parsing, conversion to numerics, and storage; instead, use placeholders:

self.conn.execute("INSERT INTO Log (Timestamp, Channel, SomePayloadBitsChanged ) VALUES (?, ?, ?)",
                  timestamp,
                  self.channel,
                  somePayloadBitsChanged
             );

Then, realize that you're really just looking for the set of indices in the input where the value is 2. Numpy makes that easy:

# this can be a static class member – it's not going to change.
SQL_QUERY = """
    INSERT INTO Log (
         Timestamp, Channel, SomePayloadBitsChanged
         )
    VALUES (?, ?, ?)
"""
def work(self, input_items, output_items):
    inp = input_items[0] # just makes life easier
    # We're looking for 2 in everything but the last 64 items,
    # because we need the 64 items following the 2
    indices = numpy.where(inp[:-64] == 2)
    if indices: # if that list weren't empty
        self.conn.execute_many(self.SQL_QUERY,
                               zip(
                               [time.time_ns()] * len(indices),
                               [self.channel] * len(indices),
                               [self.littleEndianToBig(inp[i:i+64]) for i in indices]
                               ))
    # finally, tell GNU Radio that we've worked on all the input
    # (aside from the last 64 items, see above), and want to have
    # new data, not be presented with the same items again.
    return max(0, len(inp) - 64)