dp.show() no longer lists all files

[aecorn]

Dict seems to be empty, and underdeveloped:

dapla-toolbelt/dapla/backports.py

Line 20 in b3defcc

out[trimmed_name] = {}

Consider something like this, or maybe just returning a pure list, should mimic old, backported behavior.

On request from
@ohvssb

[bjornandre]

dp.show() was originally written to recursively show all folders below a given path. So you are suggesting that it should recursively show all folders and files?

[aecorn]

I might have remembered it wrong, in that i thought it used to return a dataframe of all files in the respective folders as well.
Ive tried to dig through the old deprecated code to confirm, but the lead stopped at an old microservice that I think has been closed for business.
os.environ["CATALOG_URL"] is no longer available.
https://github.com/statisticsnorway/dapla-ipython-kernels/blob/2e1b4924b45ac0ead6ac09fc7ad6e46f7903f150/dapla/services/clients.py#L20

If my memory of what "show" should do, does not match actual history, disregard this issue.
But Id like to confirm in some way that the function actually did not list files, as that is what I remember.
Were "files" actually "folders" before? That might also be the culprit of the dissonance here....
Did dp.read_pandas() on a folder, use to, before, get the parquet file from inside the folder? But when saving with dp.write_pandas a folder was created? When we now instead just operate on "flat" files in the folder?

[bjornandre]

Yep, "datasets" were actually folders in the old "catalog" structure. In a way that can make sense, since a dataset may consist of several files. The method dp.read_pandas() still works for both files and folders - which in the latter case will read all files inside the folder into a single dataframe. This is the way spark does it.

Now we tend to operate on files directly, just like you said.

So the question is, what do we need the show method for? Maybe it should list both files and folders recursively? I've created a PR for this.

[aecorn]

Me and @ohvssb both expected it to list both files and folders a couple of days earlier, before I started digging into the history and old codebases... When the method is named something as simple as "show" you should expect a "broader behavior" perhaps?

I would expect "show()" to list both.
Alternativly you could do multiple behaviors...
show() / show(type="all")
show_files() / show(type="files")
show_folders() / show(type="folders")