Beginner - Config help needed - Kraken, Coinbase

[Antergosgeek]

Hello,

first of all thank you so much for this piece of software. It definitely fills a need that I have had for a good while now.

I am not a programmer, just a hobbyist with some basic understanding.
Need some help getting everything to work.

This is how far I have gotten:

1. Created virtualenv
2. I installed rp2 and rp2-dali
3. Ran the dali test example succesfully
4. Created Api Keys for Kraken and Coinbase
5. Copied the test_config.ini, renamed it to config.ini and adjusted it.

This is my current config:

# Global configuration

# Data loaders

[dali.plugin.input.rest.kraken USER]
account_holder = email@email.com
api_key = myapikey
api_secret = myapisecret
native_fiat = USD
thread_count = 5

I have not yet managed to succesfully setup configuration for Coinbase. The Api Key data I got for Coinbase looks different than what I expected. I got a key id and a private key of the following format:

-----BEGIN EC PRIVATE KEY-----\nlongstring==\n-----END EC PRIVATE KEY-----

So I may have done something wrong here...because this results into:

INFO: Reading crypto data using plugin 'dali.plugin.input.rest.coinbase USER'
INFO: Error <Response [401]>: b'Unauthorized\n'

What I am wondering is if I need to be using Coinbase Exchange for the API Key? I created mine in the regular Coinbase Advanced Trading User Settings --> API
So I removed it from the config.ini for now.

With only the [dali.plugin.input.rest.kraken USER] as shown above It won't run succesfully and I am not quite sure why...
Is it due to not having configured a pair converter or given a fiat access key?

When I run:
dali_us -s -o output -p test3_ config.ini

This is the log-file:

2024-07-20 08:09:12,494/dali/INFO: Country: us
2024-07-20 08:09:12,508/dali/INFO: Initialized input plugin 'dali.plugin.input.rest.kraken USER'
2024-07-20 08:09:12,514/dali/WARNING: No Fiat Access Key. Fiat pricing will NOT be available. To enable fiat pricing, an access key from exchangerate.host is required.
2024-07-20 08:09:12,517/dali/WARNING: No Fiat Access Key. Fiat pricing will NOT be available. To enable fiat pricing, an access key from exchangerate.host is required.
2024-07-20 08:09:12,517/dali/INFO: No pair converter plugins found in configuration file: using default pair converters.
2024-07-20 08:09:12,522/dali/INFO: Reading crypto data using plugin 'dali.plugin.input.rest.kraken USER'
2024-07-20 08:09:37,736/dali/INFO: Building manifest to optimize price calculation with the pair converters.
2024-07-20 08:09:37,738/dali/INFO: Resolving transactions
2024-07-20 08:09:38,317/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\dali_main.py", line 193, in _dali_main_internal
    resolved_transactions: List[AbstractTransaction] = resolve_transactions(transactions, dali_configuration, args.read_spot_price_from_web)
                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\transaction_resolver.py", line 264, in resolve_transactions
    transaction = _update_spot_price_from_web(transaction, global_configuration)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\transaction_resolver.py", line 138, in _update_spot_price_from_web
    conversion: RateAndPairConverter = _get_pair_conversion_rate(
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\transaction_resolver.py", line 110, in _get_pair_conversion_rate
    rate = cast(AbstractPairConverterPlugin, pair_converter).get_conversion_rate(timestamp, from_asset, to_asset, exchange)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\abstract_pair_converter_plugin.py", line 172, in get_conversion_rate
    historical_bar = self.get_historic_bar_from_native_source(timestamp, from_asset, to_asset, exchange)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\plugin\pair_converter\ccxt.py", line 281, in get_historic_bar_from_native_source
    if self._is_fiat_pair(from_asset, to_asset):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\abstract_pair_converter_plugin.py", line 245, in _is_fiat_pair
    return self._is_fiat(from_asset) and self._is_fiat(to_asset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\abstract_pair_converter_plugin.py", line 249, in _is_fiat
    self._build_fiat_list()
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\abstract_pair_converter_plugin.py", line 193, in _build_fiat_list
    self._check_fiat_access_key()
  File "C:\Users\USER\Python\rp2\.venv\Lib\site-packages\dali\abstract_pair_converter_plugin.py", line 343, in _check_fiat_access_key
    raise RP2ValueError(
rp2.rp2_error.RP2ValueError: No fiat access key. To convert fiat assets, please acquire an access key from exchangerate.host.The access key will then need to be added to the configuration file. For more details visit https://github.com/eprbell/dali-rp2/blob/main/docs/configuration_file.md
2024-07-20 08:09:38,345/dali/INFO: Log file: ./log/rp2_2024_07_20_08_09_10_597899.log
2024-07-20 08:09:38,345/dali/INFO: Generated output directory: output
2024-07-20 08:09:38,345/dali/INFO: Done

Is someone able to point me in the right direction?
Would really love to get this to work.

[eprbell]

Thanks for the feedback. The error you're seeing is discussed here:

dali-rp2/docs/configuration_file.md

Line 507 in 85f08f4

* The CCXT plugin makes use of the default forex exchange API, Frankfurter, which provides daily rates from the European Central Bank. Rates for bank holidays and weekends are taken from the previous trading day, so if a rate is requested for Saturday, the Friday rate will be used. Two other plugins are also available for forex - one that makes use of Exchangerate.host free plan and requires a fiat_access_key and another plugin that makes use of a CSV file for forex rates.

.

As for Coinbase, unfortunately they deprecated some REST endpoints: as a result the CB plugin no longer works and needs to be rewritten: more on this at #216). As an alternative you could use the manual plugin, which is not as easy and convenient as a native plugin, but will work.

[eprbell]

A couple of comments/questions:

• do you only have Historic Crypto as pair converter in your config file? Can you also add the Kraken pair converter?
• what version of DaLI are you using? Can you try again with the very latest (v0.6.11)?

[Antergosgeek]

Yes, so far only using Historic Crypto. Sorry that a lot of this may be very clear but I am not very used to it all. Like I said hobbyist/beginner. I can usually find my way around what I need to do but there are quite a few options here :-).
I will setup the Kraken pair converter and check my Dali version and report back.
Thank you for your patience.

[Antergosgeek]

So, I updated DaLi to v0.6.11 and added the Kraken Pair Converter. Removed the Historic Pair Converter after this run since I do not need it, was just me trying to make it work.

2024-07-27 19:56:14,762/dali/INFO: Country: us
2024-07-27 19:56:14,772/dali/INFO: Initialized input plugin 'dali.plugin.input.rest.kraken'
2024-07-27 19:56:14,772/dali/INFO: Initialized pair converter plugin 'dali.plugin.pair_converter.historic_crypto'
2024-07-27 19:56:14,782/dali/INFO: Initialized pair converter plugin 'dali.plugin.pair_converter.ccxt_kraken'
2024-07-27 19:56:14,782/dali/INFO: Reading crypto data using plugin 'dali.plugin.input.rest.kraken'
2024-07-27 19:56:39,966/dali/INFO: Building manifest to optimize price calculation with the pair converters.
2024-07-27 19:56:39,976/dali/INFO: Resolving transactions
2024-07-27 19:56:40,733/Kraken.com_CSVOHLCVT/INFO: 
In order to provide accurate pricing from Kraken, a large (4.1+ gb) zipfile needs to be downloaded.
2024-07-27 19:56:52,875/Kraken.com_CSVOHLCVT/INFO: Downloading the unified CSV from https://drive.usercontent.google.com/download?id=11WtjXA9kvVYV9KDoebGV5U75dmcA3bJa&export=download&confirm=t&uuid=bf890d46-5393-4a76-b3b0-e84c132864ae
2024-07-27 20:00:39,066/Kraken.com_CSVOHLCVT/INFO: Attempting to retrieve ATOMEUR pair from the unified Kraken CSV file.
2024-07-27 20:01:13,987/Kraken.com_CSVOHLCVT/INFO: Attempting to retrieve XMRUSDT pair from the unified Kraken CSV file.
2024-07-27 20:01:58,832/Kraken.com_CSVOHLCVT/INFO: Attempting to retrieve USDTAUD pair from the unified Kraken CSV file.
2024-07-27 20:03:14,314/Kraken.com_CSVOHLCVT/INFO: Attempting to retrieve OXTEUR pair from the unified Kraken CSV file.
2024-07-27 20:03:30,966/Frankfurter/nearest/INFO: The most accurate candle was not able to be used for pricing the asset XMR at 2024-06-22 03:08:23+00:00. 
The 4h candle for 2024-06-22 04:00:00+00:00 was used. 
The price may be inaccurate. If you feel like you're getting this message in error, 
please open an issue at https://github.com/eprbell/dali-rp2/issues
2024-07-27 20:03:43,542/Frankfurter/nearest/INFO: The most accurate candle was not able to be used for pricing the asset XMR at 2024-05-18 17:17:35+00:00. 
The 1d candle for 2024-05-19 00:00:00+00:00 was used. 
The price may be inaccurate. If you feel like you're getting this message in error, 
please open an issue at https://github.com/eprbell/dali-rp2/issues
2024-07-27 20:03:44,556/dali/INFO: Generating config file in output
2024-07-27 20:03:44,556/dali/INFO: Generating input file in output
2024-07-27 20:03:44,586/dali/INFO: Generated output: C:\Users\USER\Python\rp2\output\test4_crypto_data.ods
2024-07-27 20:03:44,586/dali/INFO: Log file: ./log/rp2_2024_07_27_19_56_14_092405.log
2024-07-27 20:03:44,586/dali/INFO: Generated output directory: output
2024-07-27 20:03:44,586/dali/INFO: Done

Looks like that did the trick, now I just need to rightly interpret the files, have a few __unknown under Unique ID as well as some under Crypto Received and Crypto Sent from transferring in/out from/to software wallets but that I think I can figure out. At least the automatic import and file creation with DaLi seems to work. Just need to check on that 2024-07-27 20:03:30,966/Frankfurter/nearest/INFO: The most accurate candle was not able to be used for pricing the asset XMR at 2024-06-22 03:08:23+00:00. and those __unknown.

Thank you for helping me to this point!

[eprbell]

No worries! The topic of __unknown in transactions is dealt with in this FAQ: https://github.com/eprbell/dali-rp2/blob/main/docs/user_faq.md#what-if-a-transaction-is-generated-with-some-fields-set-to-__unknown.

For the other issue, CC @macanudo527, who may have more insight on that topic.

[Antergosgeek]

Thank you so much!