-
|
I'm trying to configure the civitai extractor a very specific way but I can't figure out how to do it. I've spent many hours trying and since then a lot of images have been purged (and more images get purged as rules get updated over time) model1/model2, etc should have the name I don't know if it's possible to have no model version so it might need to fallback to a default name or something My WIP config is a mess "civitai": {
"#archive": "postgresql://user:pass@192.168.1.111/gallery-dl",
"archive-prefix": "civitai.com, ",
"archive-format": "{id}",
"archive-pragma": [
"journal_mode=WAL",
"synchronous=NORMAL"
],
"archive-mode": "file",
"archive-table": "civitai.com-media",
"archive-event": [
"file",
"skip"
],
"base-directory": "$OTHER/civitai.com/",
"api": "trpc",
"api-key": REDACTED,
"retries": 100,
"files": [
"image",
"model"
],
"include": [
"user-models",
"#user-posts",
"#user-images",
"#user-videos"
],
"metadata": [
"generation"
],
"sleep-request": 0,
"nsfw": 15,
"quality": "original=true",
"search": {
"directory": [
"search",
"{user[username}"
]
},
"user-models": {
"directory": [
"users",
"{username|user[username]}",
"models",
"{model[name]}"
],
"category-transfer": true,
"filename": "{file[name]}"
},
"user-posts": {
"directory": [
"users",
"{username|user[username]}",
"posts",
"{post[id]}{post[title]:? //}"
],
"category-transfer": true
},
"user-images": {
"directory": [
"users",
"{username|user[username]}",
"images",
"{model[name]}"
],
"category-transfer": true
},
"user-videos": {
"directory": [
"users",
"{username|user[username]}",
"videos",
"{model[name]}"
],
"category-transfer": true
},
"user": {
"category-transfer": false
},
"postprocessors": [
{
"name": "classify",
"filter": "subcategory == 'user-models'",
"mapping": {
"images": [
"bmp",
"exr",
"hdr",
"j2k",
"jp2",
"jpe",
"jpeg",
"jpg",
"jxr",
"png",
"tif",
"tiff",
"webp"
],
"model": ""
},
"directory": [
"users",
"{username|user[username]}",
"models",
"model[name]"
]
},
{
"name": "metadata",
"mode": "json",
"extension": "json",
"extension-format": "{extension!l}.json",
"event": "file",
"mtime": true
}
]
}My goal is to save all types of content from users that I want to save content from. |
Beta Was this translation helpful? Give feedback.
Answered by
mikf
May 19, 2025
Replies: 2 comments 15 replies
-
|
No, I don't think this actually is currently possible without some custom and possibly quite involved At least the directory split into The The "best" that I can come up with is the following, but your config more or less already accomplishes that: edit 1: edit 2: {
"extractor": {
"civitai": {
"user": {
"include": ["user-images", "user-models"]
},
"user-images": {
"metadata": "version",
"directory": [
"users",
"{user[username]}",
"images",
"{model[name]}"
]
},
"user-models": {
"category-transfer": true,
"postprocessors": "directory",
"files": ["image", "model"],
"directory": {
"extension == 'bin'": [
"users",
"{user[username]}",
"models",
"{model[name]}",
"{version[name]}",
"model"
],
"": [
"users",
"{user[username]}",
"models",
"{model[name]}",
"{version[name]}",
"images"
]
}
}
}
}
} |
Beta Was this translation helpful? Give feedback.
14 replies
-
|
The link that errors has "/api/trpc/" in the url. I don't know if that's the actual download link or not, but that erroring is preventing it from using the real download link if it isn't. [civitai][error] HttpError: '500 Internal Server Error' for 'https://civitai.com/api/trpc/image.getInfinite?input=%7B%22json%22%3A%7B%22modelVersionId%22%3A339063%2C%22prioritizedUserIds%22%3A%5B2014187%5D%2C%22period%22%3A%22AllTime%22%2C%22sort%22%3A%22Most+Reactions%22%2C%22limit%22%3A20%2C%22pending%22%3Atrue%2C%22cursor%22%3A6525283%7D%7D' |
Beta Was this translation helpful? Give feedback.
-
|
I'm getting this error. I don't know how to make it fallback to something else if there is no value (unless if I'm misunderstanding what this error is about): I didn't get to downloading more until now because I wasn't sure if you were going to say something about the last thing I said so I waited (it's fine if you don't, I just didn't know). |
Beta Was this translation helpful? Give feedback.
-
|
I did a lot of testing and the only thing that fixes it is removing |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for taking this long to respond. I've had (and to some extend still have) to deal with health issues myself ... This exception happens when |
Beta Was this translation helpful? Give feedback.
Answer selected by
Twi-Hard
-
|
I'm sorry, I made the issue not knowing you replied. I thought I checked every email since I started this Q&A. Thank you very much :) |
Beta Was this translation helpful? Give feedback.
-
|
Not at all. Reporting errors is always(*) OK, especially if it was my fault. (*) Unless it is the 10th iteration of the same error report and there is already a pinned issue for it. That is annoying. |
Beta Was this translation helpful? Give feedback.
-
This has no effect when using a PostgreSQL archive. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Also, pairing |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Sorry for taking this long to respond. I've had (and to some extend still have) to deal with health issues myself ...
This exception happens when
versionmetadata is enabled and an image doesn't have any version info, i.e. nomodelVersionId. Should be fixed in a6de87d