Author Archives: proprietor

Backtrader Reference Info

I found the following to be useful when working with the Backtrader testing/trading engine.

To get “self” variables from a strategy for inspection after the run is over:

...
strategies = cerebro.run()
...
strat = strategies[0]
# the strategy uses self.hypo_long_orders as a variable
strat.hypo_long_orders

To view orders from the system this approach should work:

# orders
orders = [str(order).splitlines() for order in strat._orders]
df = pd.DataFrame(orders)
df

#trades
trades = [str(trade).splitlines() for trade in list(strat._trades.values())[0][0]]
df1 = pd.DataFrame(trades)
df1

#raw data
# raw data
open = strat.data_open.array
high = strat.data_high.array

I adapted some of this from the following place in the backtrader community:

https://community.backtrader.com/topic/809/accessing-data-after-finished-run/6

With respect to using backtrader with IB the following appears to be useful:

https://community.backtrader.com/topic/1203/is-it-possible-to-retrieve-all-the-open-active-orders-from-interactive-broker

https://community.backtrader.com/topic/1146/is-there-way-to-track-open-orders-with-ib-broker

https://medium.com/@danjrod/interactive-brokers-in-python-with-backtrader-23dea376b2fc

https://community.backtrader.com/user/richard-o-regan

https://pypi.org/project/ibapi/

https://medium.com/codex/using-python-to-send-telegram-messages-in-3-simple-steps-419a8b5e5e2

Backtrader Order Execution Sequencing

I’ve been working on a project that has many orders pending in the system at a given time. I’ve been noticing a difference from the fills I get from my broker and the hypothetical fills I get from the backtrader testing engine. I think I’ve finally gotten to the bottom of what is going on, although I’m not sure how to fix it.

I use the following code to show the orders that are active in the system and show when an order is filled or cancelled due to margin concerns:

    def show_open(self):
        print('open orders from show orders function: ')
        print('^' * 50)
        long_orders = []
        short_orders = []
        raw_orders= self.broker.get_orders_open()
        #print(format(raw_orders))
        for _ in self.broker.get_orders_open(): #cerebro.broker.orders:
            
            raw = _
            ref = _.ref
            alive = _.alive()
            long = _.isbuy()
            osize = _.created.size
            oprice = _.created.price
            if long == True:
                long_orders.append({'ref': ref,'long': long, 'size': osize, 'price': oprice})
            else:
                short_orders.append({'ref': ref,'long': long, 'size': osize, 'price': oprice})
            
        long_orders = sorted(long_orders,key = lambda d: d['price'],\
                                  reverse = True)
        short_orders = sorted(short_orders, key = lambda d: d['price'])

        for l in long_orders:
            print('{},{},{},{}' . format(l['ref'],l['long'],l['size'],round(l['price'],5)))
        for s in short_orders:
            print('{},{},{},{}' . format(s['ref'],s['long'],s['size'],round(s['price'],5)))
            #print(raw) 
        print('end_show_open_function____' * 3)     


    def notify_order(self, order):
        if order.status in [order.Completed]:
            print('{} Order Completed at {}' . format(order.ref, round(order.created.price,5)))
            
        if order.status in [order.Canceled, order.Margin, order.Rejected]:
            
            print('{} Order Canceled/Margin/Rejected Price: {}' .format(order.ref, round(order.created.price,5)))
        pass

These methods are part of a backtrader strategy class. In the system in question I have the following output

2022-09-08 06:30:00 buy size limit 0.9988 close 1.00197 position 0
open orders from show orders function: 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
82,True,15000,0.9986
80,True,15000,0.99785
81,True,15000,0.99785
56,True,90000,0.99745
51,True,90000,0.9974
55,True,90000,0.99735
54,True,90000,0.9973
57,True,90000,0.99725
52,True,90000,0.9972
78,True,15000,0.9969
53,True,90000,0.9968
73,True,90000,0.9967
71,True,90000,0.99665
79,True,15000,0.99665
77,True,15000,0.99655
59,True,90000,0.99645
70,True,90000,0.99645
68,True,90000,0.9963
47,True,90000,0.99615
50,True,90000,0.9961
58,True,90000,0.99605
46,True,90000,0.9959
69,True,90000,0.99585
60,True,90000,0.9958
62,True,90000,0.99575
64,True,90000,0.99565
74,True,90000,0.99565
66,True,90000,0.9956
65,True,90000,0.99555
48,True,90000,0.99545
67,True,90000,0.99545
76,True,15000,0.99535
61,True,90000,0.99525
63,True,90000,0.99525
75,True,15000,0.995
49,True,90000,0.9948
45,True,90000,0.9939
44,True,90000,0.99185
41,True,90000,0.99145
42,True,90000,0.9911
43,True,90000,0.99055
40,True,90000,0.99025
29,True,15000,0.9875
32,True,15000,0.9874
2,True,15000,0.9873
3,True,15000,0.98715
7,True,15000,0.98715
8,True,15000,0.98715
5,True,15000,0.9871
4,True,15000,0.987
6,True,15000,0.9868
30,True,15000,0.9868
24,True,15000,0.9867
11,True,15000,0.98665
9,True,15000,0.98655
33,True,15000,0.98655
22,True,15000,0.9865
26,True,15000,0.98645
10,True,15000,0.9863
23,True,15000,0.9863
21,True,15000,0.9862
14,True,15000,0.986
13,True,15000,0.98595
15,True,15000,0.98585
16,True,15000,0.9858
12,True,15000,0.9857
19,True,15000,0.9856
17,True,15000,0.98535
18,True,15000,0.9851
20,True,15000,0.9851
end_show_open_function____end_show_open_function____end_show_open_function____
82 Order Completed at 0.9986
83 Order Completed at 0.9988

A few bars later the following occurs:

2022-09-08 07:30:00 sell size limit 1.0033 close 0.99998 position 30000
open orders from show orders function: 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
80,True,15000,0.99785
81,True,15000,0.99785
56,True,90000,0.99745
51,True,90000,0.9974
55,True,90000,0.99735
54,True,90000,0.9973
57,True,90000,0.99725
52,True,90000,0.9972
78,True,15000,0.9969
53,True,90000,0.9968
73,True,90000,0.9967
71,True,90000,0.99665
79,True,15000,0.99665
77,True,15000,0.99655
59,True,90000,0.99645
70,True,90000,0.99645
68,True,90000,0.9963
47,True,90000,0.99615
50,True,90000,0.9961
58,True,90000,0.99605
46,True,90000,0.9959
69,True,90000,0.99585
60,True,90000,0.9958
62,True,90000,0.99575
64,True,90000,0.99565
74,True,90000,0.99565
66,True,90000,0.9956
65,True,90000,0.99555
48,True,90000,0.99545
67,True,90000,0.99545
76,True,15000,0.99535
61,True,90000,0.99525
63,True,90000,0.99525
75,True,15000,0.995
49,True,90000,0.9948
45,True,90000,0.9939
44,True,90000,0.99185
41,True,90000,0.99145
42,True,90000,0.9911
43,True,90000,0.99055
40,True,90000,0.99025
29,True,15000,0.9875
32,True,15000,0.9874
2,True,15000,0.9873
3,True,15000,0.98715
7,True,15000,0.98715
8,True,15000,0.98715
5,True,15000,0.9871
4,True,15000,0.987
6,True,15000,0.9868
30,True,15000,0.9868
24,True,15000,0.9867
11,True,15000,0.98665
9,True,15000,0.98655
33,True,15000,0.98655
22,True,15000,0.9865
26,True,15000,0.98645
10,True,15000,0.9863
23,True,15000,0.9863
21,True,15000,0.9862
14,True,15000,0.986
13,True,15000,0.98595
15,True,15000,0.98585
16,True,15000,0.9858
12,True,15000,0.9857
19,True,15000,0.9856
17,True,15000,0.98535
18,True,15000,0.9851
20,True,15000,0.9851
84,False,-30000,1.0043
end_show_open_function____end_show_open_function____end_show_open_function____
45 Order Canceled/Margin/Rejected Price: 0.9939
46 Order Canceled/Margin/Rejected Price: 0.9959
47 Order Canceled/Margin/Rejected Price: 0.99615
48 Order Canceled/Margin/Rejected Price: 0.99545
49 Order Canceled/Margin/Rejected Price: 0.9948
50 Order Canceled/Margin/Rejected Price: 0.9961
51 Order Canceled/Margin/Rejected Price: 0.9974
52 Order Canceled/Margin/Rejected Price: 0.9972
53 Order Canceled/Margin/Rejected Price: 0.9968
54 Order Canceled/Margin/Rejected Price: 0.9973
55 Order Canceled/Margin/Rejected Price: 0.99735
56 Order Canceled/Margin/Rejected Price: 0.99745
57 Order Canceled/Margin/Rejected Price: 0.99725
58 Order Canceled/Margin/Rejected Price: 0.99605
59 Order Canceled/Margin/Rejected Price: 0.99645
60 Order Canceled/Margin/Rejected Price: 0.9958
61 Order Canceled/Margin/Rejected Price: 0.99525
62 Order Canceled/Margin/Rejected Price: 0.99575
63 Order Canceled/Margin/Rejected Price: 0.99525
64 Order Canceled/Margin/Rejected Price: 0.99565
65 Order Canceled/Margin/Rejected Price: 0.99555
66 Order Canceled/Margin/Rejected Price: 0.9956
67 Order Canceled/Margin/Rejected Price: 0.99545
68 Order Canceled/Margin/Rejected Price: 0.9963
69 Order Canceled/Margin/Rejected Price: 0.99585
70 Order Canceled/Margin/Rejected Price: 0.99645
71 Order Canceled/Margin/Rejected Price: 0.99665
73 Order Canceled/Margin/Rejected Price: 0.9967
74 Order Canceled/Margin/Rejected Price: 0.99565
75 Order Completed at 0.995
76 Order Completed at 0.99535
77 Order Completed at 0.99655
78 Order Completed at 0.9969
79 Order Canceled/Margin/Rejected Price: 0.99665
80 Order Canceled/Margin/Rejected Price: 0.99785
81 Order Canceled/Margin/Rejected Price: 0.99785

Trades 79, 81 and 80 are closest to the market but are cancelled due to margin while order 75, 76, 77, and 78 are filled. If all the orders were pending with a broker this is most certainly NOT what would have happened; the orders most closest to the market would have been filled and then the rest of the orders would have been cancelled due to margin.

It doesn’t appear that backtrader looks at the trades closest to the market when determining which orders to fill. I’m not certain how the algorithm works, but I don’t see how the simulated results could possibly have been obtained in live trading.

It appears the cerebro engine goes through sequentially by order number. In this case some of the orders were for 90k, which would have caused a margin violation so they were cancelled. The system then hit order 75, which was entered prior to orders 79, 80, and 81 so filled it first even though by market action the 79, 80, and 81 should have been filled first.

On this post on the backtrader community the admin seems to indicate that trades are based on a FIFO just like on an exchange. I don’t think this is really how exchanges work though in my use case.

https://community.backtrader.com/topic/1933/buy-and-sell-orders-execution-order