实现一个随机图api

RealXLFD2024-10-272024-10-28

浅谈一下2023.7-2024.11这段时间做的一个小事情

关于这一年的我和博客

想起来，博客建立至今也已经有一年多的时间了。当然对于我这种懒狗肯定是建好了懒得维护然后就作废的…
我不知道写什么东西放上去
也不知道怎么写。
所以除了一开始放的那几篇之后几乎从未更新
那时候我连编程都不会，更别提产出啥有价值的东西了
但是，一年时间过去…我选择重新拾起这个博客
删掉之前写的垃圾
然后记录一下一些我想写的东西

起点

目标很简单，开发（copy）一个博客，因为无聊
每天大叫完还有大把的时间，大一暑假，为何不拿这些时间做点闲事呢？
于是我在网上瞎逛，发现了这个博客主题（anzhiyu，butterfly魔改)
一眼爱上
于是开始学习
看文档
…
需求很简单: 随机背景图
看到这个需求，脑袋里最先浮现出的解法是：

const pics = [
    "https://a.com/1.png",
    "..."
]
const r_num = Math.random() * 10000;
const target_url = rpics[r_num % pics.length]

但是我肯定不会这么写，毕竟麻烦
假设有1w张图片呢？那光是js都要占很多空间，难以维护，拖慢网站加载速度
那我们能不能写个后端解决这个问题呢？
不想维护+低成本
那cloudflare workers就成为了首选
用不完的免费workers额度 + 10GB r2 bucket空间 + 数一数二的防攻击
于是，初版完成了:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})
const R2 = r2_realxlfd // 对应你储存桶的变量名
const KV = kv_realxlfd // 对应你KV 耐用对象的变量名

// refresh pics in current URL
async function refreshPictrues(directoryPath) {
  const listed = await R2.list({ prefix: 'randompics' + directoryPath }); // 你可以将'randompics'改成你自己喜欢的文件夹名字
  let truncated = listed.truncated;
  let cursor = truncated ? listed.cursor : undefined;
  let pic_queue_temp = []
  while (truncated) {
    const next = await R2.list({
      prefix: 'randompics' + directoryPath,
      cursor: cursor,
    });
    listed.objects.push(...next.objects);
    truncated = next.truncated;
    cursor = next.cursor
  }
  for (let obj of listed.objects) {
    // Make sure the format of file is picture
    if (/\.(webp|jpg|jpeg|png)$/.test(obj.key))
      pic_queue_temp.push(obj.key)
  }
  if (pic_queue_temp.length !== 0) {
    await KV.put(directoryPath, JSON.stringify(pic_queue_temp))
  }
  return pic_queue_temp
}
// Purge all KV cache
async function KV_deleteAll() {
  const keys = await KV.list();
  await Promise.all(keys.keys.map(async (key) => {
    await KV.delete(key.name);
    console.log('delete ' + key.name)
  }));
}

async function handleRequest(request) {
  let picsobj = []
  // Get the access URL
  const url = new URL(request.url)
  let currentPath = url.pathname
  // trigger purge
  if (currentPath === '/purge') {
    await KV_deleteAll()
    return new Response('Purge Success', status: 404
    });
  }
  // 从KV中获取缓存，没有缓存，则生成缓存
  const str_picobj = await KV.get(currentPath);
  if (str_picobj === null) {
    picsobj = await refreshPictrues(currentPath)
  }
  else {
    picsobj = JSON.parse(str_picobj)
  }
  const picReturn = picsobj[Math.floor(Math.random() * picsobj.length)]
  // 生成响应
  const response = await R2.get(picReturn)
  console.log(picReturn)
  if (response === null) {
    refreshPictrues(currentPath)
    return new Response('Image Not Found', {
      status: 404
    })
  }
  return new Response(response.body, {
    headers: {
      'content-type': 'image/webp' // 这里还有问题，但是刚学js不久的我很菜没注意到
    }
  })
}

具体有什么特点呢？

基于r2 bucket 路径的文件分类
简单来说，假设我上传了/albumA/1.png和/albumB/2.png
那么访问https://.../albumA就可以随机返回1.png或者该文件夹下的一张其他图片
albumB同理
自动索引，手动触发索引更新
访问/purge则会遍历配置的目录，将其所有子目录中的所有图片路径载入到数据索引中
或者请求到一张索引中有但是未在r2 bucket中找到文件的图片，也会触发索引更新
当时cf d1还没出来，所以使用的是json + cf KV

也就是说，实际上想要添加图片，只需要我们把图片文件上传到r2 储存桶中去就行了

新的需求，新的挑战

偶然的一个机会，看到了某个陌生的博客
被首页的大图深深吸引
为什么会有渐变效果呢？
omg
原来是先请求一张小图（缩略图）再请求大图来缩短响应时间，使用模糊来规避小图导致的图片不清晰，并且使用从模糊到清晰的渐变流利的从小图切换到大图

具体来说：缩略图大概100kb不到，响应速度快，载入速度快，先载入小图再替换为大图，可以显著减少白屏时间
nb
看到这么牛逼的东西，我肯定要copy过来
毕竟信仰拿来主义
别人的东西，抄过来就是我的
于是copy过来了
调渐变效果调了我一年，真的是煎熬…感觉每一ms的timeout，每1px的blur都给我试了一遍
终于舒服了
但是…随机图后端呢？

需求

随机图中每一张图需要一张小图，一张大图
需要通过某种方式，能够确保返回的大图和小图是同一张图片

怎么办

V2.0

大图小图的问题：
创建2个文件夹，一个装大图，一个装小图。

1 2	large/albumA/big_1.png small/albumA/small_1.png

添加咨询参数?l返回大图，?s返回小图
数据索引只索引后面的相对路径部分(albumA/xxx.png)，所以只需要拼接文件夹路径即可得到完整路径

返回同一张图片的问题：
rid
用rid来解决问题。rid=前端生成的随机种子id，可以无限大
也就是说，将生成随机数的职责交给了前端。前端生成一个随机数种子，后端获得了该种子，能够确保在数据索引发生变化之前指向同一张图片
也就是

1 2	let index = [...] let target_url = index[rid % index.length]

开搞

// src/proc/refresh.js
async function refreshPictures(directoryPath, env) {
  const listed = await env.r2_realxlfd.list({ prefix: "sites" + directoryPath });
  let truncated = listed.truncated;
  let cursor = truncated ? listed.cursor : void 0;
  let pic_queue_temp = [];
  while (truncated) {
    const next = await env.r2_realxlfd.list({
      prefix: "randompics" + directoryPath,
      cursor
    });
    listed.objects.push(...next.objects);
    truncated = next.truncated;
    cursor = next.cursor;
  }
  for (let obj of listed.objects) {
    if (/\.(webp|jpg|jpeg|png)$/.test(obj.key))
      pic_queue_temp.push(obj.key);
  }
  if (pic_queue_temp.length !== 0) {
    await KV.put(directoryPath, JSON.stringify(pic_queue_temp));
  }
  return pic_queue_temp;
}
async function deleteAll(env) {
  const keys = await env.kv_realxlfd.list();
  await Promise.all(
    keys.keys.map(async (key) => {
      await env.kv_realxlfd.delete(key.name);
      console.log("delete " + key.name);
    })
  );
  const mainTone_json = await env.r2_realxlfd.get("functions/mainTone.json");
  await env.kv_realxlfd.put("mainTone", mainTone_json.body);
}

// src/proc/response.js
async function purge() {
  const page_purge = await fetch(
    "https://.../res/html/purge.html"
  );
  return new Response(await page_purge.text(), {
    headers: {
      "Content-Type": "text/html"
    },
    status: 404
  });
}
async function errPage() {
  const page_404 = await fetch(
    "https://.../res/html/404.html"
  );
  return new Response(await page_404.text(), {
    headers: {
      "Content-Type": "text/html"
    },
    status: 404
  });
}
function mainColor(jsonText) {
  return new Response(JSON.stringify(jsonText), {
    headers: {
      "Access-Control-Allow-Origin": origin,
      "Access-Control-Allow-Methods": "GET, POST, OPTIONS",
      "Access-Control-Allow-Headers": "Content-Type",
      "Content-Type": "application/json"
    }
  });
}
function streamResp(response, fileType, origin2) {
  const stream = new ReadableStream({
    start(controller) {
      const reader = response.body.getReader();
      function read() {
        reader.read().then(({ done, value }) => {
          if (done) {
            controller.close();
            return;
          }
          controller.enqueue(value);
          read();
        });
      }
      read();
    }
  });
  return new Response(stream, {
    headers: {
      "Access-Control-Allow-Origin": origin2,
      "Access-Control-Allow-Methods": "GET, POST, OPTIONS",
      "Access-Control-Allow-Headers": "Content-Type",
      "content-type": fileType
    }
  });
}

// src/index.js
var src_default = {
  async fetch(request, env, ctx) {
    return handleRequest(request, env);
  }
};
async function handleRequest(request, env) {
  const R2 = env.r2_realxlfd, KV2 = env.kv_realxlfd;
  let picReturn, response, fileType, picsobj;
  const url = new URL(request.url);
  const url_parts = url.pathname.split("!");
  let currentPath = url_parts[0];
  let params = url.search.slice(1);
  if (url_parts[1])
    params = url_parts[1];
  if (currentPath === "/purge") {
    await deleteAll();
    return purge();
  }
  const referer = request.headers.get("Referer");
  const origin2 = request.headers.get("Origin");
  const str_picobj = await KV2.get(currentPath);
  if (str_picobj === null) {
    picsobj = await refreshPictures(currentPath, env);
  } else {
    picsobj = JSON.parse(str_picobj);
  }
  let compress = false;
  while (true) {
    if (params !== "" && params !== null) {
      let letters, numbers;
      const matches = params.match(/([a-zA-Z]+)(\d+)/);
      if (matches) {
        letters = matches[1];
        numbers = matches[2];
        if (numbers === void 0) {
          return errPage();
        }
      }
      const formatted_number = parseInt(numbers) % picsobj.length;
      if (letters === "b") {
        picReturn = picsobj[formatted_number];
      } else if (letters === "s") {
        picReturn = picsobj[formatted_number];
        compress = true;
      } else if (letters === "c") {
        const filename = currentPath + "/" + picsobj[formatted_number].split("/").pop().split(".")[0];
        const mainTone_json_cache = await KV2.get("mainTone");
        const search_json_obj = JSON.parse(mainTone_json_cache);
        console.log(filename);
        const response2 = {
          RGB: "#" + search_json_obj[filename].replace("#", "")
        };
        return mainColor(response2);
      }
    } else {
      picReturn = picsobj[Math.floor(Math.random() * picsobj.length)];
    }
    const fileExtension = picReturn.split(".").pop().toLowerCase();
    fileType = checkType(fileExtension);
    if (fileType === "unknown") {
      R2.delete(picReturn);
      continue;
    }
    if (compress) {
      response = await R2.get(picReturn.replace("sites", "sites_compressed"));
      if (response === null) {
        await refreshPictures(currentPath, env);
        continue;
      }
    } else {
      response = await R2.get(picReturn);
      if (response === null) {
        await refreshPictures(currentPath, env);
        continue;
      }
    }
    break;
  }
  return streamResp(response, fileType, origin2);
}
function checkType(type) {
  switch (type) {
    case "webp":
      return "image/webp";
    case "png":
      return "image/png";
    case "jpg":
    case "jpeg":
      return "image/jpeg";
    default:
      return "unknown";
  }
}
export {
  src_default as default
};
//# sourceMappingURL=index.js.map

特点

流式响应，避免完整地将图片写入worker内存，加快响应速度
使用?s<rid>的参数指明了大小与rid
referer鉴权

为啥要鉴权呢？因为朋友发现一个一个一个

一个sb

我的天copy博客源码就算了，还把我的随机图api链接给copy了

这么能超一定是铁牛子把

因为太过逆天，于是直接立马加上了鉴权

加了之后大概几小时之后那位的博客就换链接了

不过我也能理解

为什么呢？因为我也是懒狗+煞笔

要我能抄我也抄，嘻嘻

一个小小的问题

使用?k=v这样的标准咨询参数，有的浏览器会缓存请求，于是兼容适配了将?替换为!的请求形式，其他不变

代价是什么呢？

用的时候爽，上传的时候可就遭殃了
就跟BT下载一样，下载的人满为患，做种的就那么几个，特别是冷门的东西

每次到上传的时候，都需要提前为每个图片准备好压缩+缩放后的小图
然后分类存放
上传的复杂度直接几何级增长
咋办

对不起每次遇到这种b情况脑子里想到的都是脚本

那时候我连python都不会 (当然现在更不会，我觉得python是世界上最难学的编程语言)
但是脑子里只想着快速解决问题
于是现学（抄）现卖
python写（抄）了一个上传脚本（chatgpt+google)
依稀记得用了PIL库压缩图片，然后好像是用rclone上传同步
我找找有没有记录

好吧这是一个批量图片压缩工具：

from PIL import Image
import concurrent.futures
import os
import threading
import shutil
import json
from alive_progress import alive_bar
from termcolor import colored
from halo import Halo
import time
import logging
import re

Image.MAX_IMAGE_PIXELS = 23000000000
# LOAD CONFIG
start_time = 0
appConfig = {
    "save": {"format": "webp", "width": 2560, "quality": 90},
    "check": {"type": ["jpg", "png", "jpeg", "webp"]},
    "dir": {"working": "./", "save": "COMPRESSED", "error": "ERROR"},
    "work": {"process": 10},
}


def printConfig():
    for key in appConfig:
        print(colored("------ " + key + " ------", attrs=["bold"]))
        for key2 in appConfig[key]:
            print(
                colored(key2, "green"),
                ":",
                colored(appConfig[key][key2], "light_blue"),
            )
    print(colored("-----------------", "white"))


def readConfig():
    global appConfig
    try:
        with open("config.json", "r") as file:
            json_data = file.read()
            appConfig = json.loads(json_data)
        print(colored(">> Config Loaded <<", "light_green"))
        printConfig()
        return
    except Exception as e:
        print(
            colored("ConfigLoadError:", "yellow"),
            e,
            colored("\ndefault config loaded.", "green"),
        )
        printConfig()
        with open("config.json", "w", encoding="utf-8") as file:
            json.dump(appConfig, file, indent=4, ensure_ascii=False)
        return


# GENERATE ALGORITHM/FORMAT FIT
config_save_quality = int(appConfig["save"]["quality"])
img_algorithm = Image.LANCZOS
if config_save_quality < 80 and config_save_quality > 50:
    img_algorithm = Image.BILINEAR
elif config_save_quality <= 50:
    img_algorithm = Image.NEAREST

config_save_format = appConfig["save"]["format"]
if appConfig["save"]["format"] == "jpg":
    config_save_format = "jpeg"
# CREATE OBJECT
spinner_load = Halo(text="Loading")
spinnerMessages = {
    "start": "Scanning Image Files",
    "finish": colored("Success!", "green"),
    "error": colored("Invaild Image Found!", "red"),
}

# LOG
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s",
    filename="log.txt",
    encoding="utf-8",
)


def removeTextColor(text):
    pattern = re.compile(r"\x1b\[\d+m")
    return re.sub(pattern, "", text)


def addLog(*args):
    log = " ".join(str(arg) for arg in args)
    print(log)
    logging.error(removeTextColor(log))


# ADD THREADLOCK
print_lock = threading.Lock()


# SCANFILES
class ProgressCache:
    def __init__(self):
        self.print_lock = threading.Lock()
        self.task_pool_com = []
        self.task_pool_copy = []
        self.task_error = []


progress_cache = ProgressCache()


def scanFiles():
    global progress_cache,start_time
    print("\n" + colored("Please input PATH", "light_magenta"), "> ", end="")
    input_dir = input()
    if input_dir == "":
        input_dir = appConfig["dir"]["working"]
    elif not os.path.exists(input_dir):
        print(colored("Your input PATH is not valid!", "red"))
        return False
    os.chdir(input_dir)
    start_time = int(time.time() * 1000)
    spinner_load.start(spinnerMessages["start"])
    exist_file_count = 0
    for root, dir, filenames in os.walk(input_dir):
        if appConfig["dir"]["save"] in root:
            continue
        elif appConfig["dir"]["error"] in root:
            continue
        for file in filenames:
            origin_filepath = os.path.join(root, file)
            save_filepath = os.path.join(
                input_dir,
                appConfig["dir"]["save"],
                os.path.relpath(root, input_dir),
                os.path.splitext(file)[0] + "." + appConfig["save"]["format"],
            )
            if os.path.exists(save_filepath):
                exist_file_count += 1
                continue
            file_ext = os.path.splitext(file)[1][1:].lower()
            if file_ext not in appConfig["check"]["type"]:
                save_filepath_copy = os.path.join(
                    input_dir,
                    appConfig["dir"]["save"],
                    os.path.relpath(root, input_dir),
                    file,
                )
                progress_cache.task_pool_copy.append(
                    (origin_filepath, save_filepath_copy)
                )
                continue
            if not verifyImg(origin_filepath):
                progress_cache.task_error.append(origin_filepath)
                continue
            progress_cache.task_pool_com.append((origin_filepath, save_filepath))
    if progress_cache.task_error == []:
        spinner_load.succeed(spinnerMessages["finish"])
        addLog("Exist", colored(str(exist_file_count), "yellow"), "Files, Skipped")
        time.sleep(1)
        return True
    else:
        spinner_load.fail("Error!")
        handleScanedErrorImg(input_dir)
        return True


def verifyImg(ori_filepath):
    bValid = True
    try:
        Image.open(ori_filepath).verify()
    except:
        bValid = False
    return bValid


# HANDLE FILES
def handleScanedErrorImg(work_dir):
    os.makedirs(os.path.join(work_dir, appConfig["dir"]["error"]), exist_ok=True)
    for error_file in progress_cache.task_error:
        file_basename = os.path.basename(error_file)
        rel_path = os.path.relpath(error_file, work_dir)
        shutil.copy(
            error_file, os.path.join(work_dir, appConfig["dir"]["error"], file_basename)
        )
        addLog(colored(rel_path, "red"))


def handleTaskPool():
    copy_task_num = len(progress_cache.task_pool_copy)
    com_task_num = len(progress_cache.task_pool_com)
    if copy_task_num != 0:
        with alive_bar(
            copy_task_num, bar="classic", title=colored("  COPY  ", "light_cyan")
        ) as bar:
            for task in progress_cache.task_pool_copy:
                try:
                    ori_file, save_file = task[0], task[1]
                    directory, _ = os.path.split(save_file)
                    os.makedirs(directory, exist_ok=True)
                    shutil.copy(ori_file, save_file)
                    bar()
                except:
                    with print_lock:
                        addLog(colored("CopyError:", "yellow"), colored(task[0], "red"))
    if com_task_num != 0:
        with alive_bar(
            com_task_num, bar="classic", title=colored("COMPRESS", "light_cyan")
        ) as bar:
            with concurrent.futures.ThreadPoolExecutor(
                max_workers=int(appConfig["work"]["process"])
            ) as executor:
                thread_futures = []
                for task in progress_cache.task_pool_com:
                    thread_future = executor.submit(handleImg, task[0], task[1])
                    thread_futures.append(thread_future)
                for future in concurrent.futures.as_completed(thread_futures):
                    bar()
    else:
        with print_lock:
            print(colored("No Image Need to Compress!", "yellow"))
    return


def handleImg(ori_filepath, save_filepath):
    try:
        with Image.open(ori_filepath) as load_img:
            directory, _ = os.path.split(save_filepath)
            os.makedirs(directory, exist_ok=True)
            img_width, img_height = load_img.size[0], load_img.size[1]
            config_save_width = int(appConfig["save"]["width"])
            if config_save_width < img_width:
                new_height = int(config_save_width * img_height / img_width)
                load_img.resize((config_save_width, new_height), img_algorithm).save(
                    save_filepath,
                    config_save_format,
                    quality=config_save_quality,
                )
            else:
                load_img.save(
                    save_filepath,
                    config_save_format,
                    quality=config_save_quality,
                )
        return
    except Exception as e:
        relpath = os.path.relpath(ori_filepath, appConfig["dir"]["working"])
        with print_lock:
            addLog(colored("CompressError:", "yellow"), colored(str(relpath), "red"))
        return


# MAIN
def main():
    try:
        readConfig()
        while True:
            if scanFiles():
                handleTaskPool()
                end_time = int(time.time() * 1000)
                print(colored(">> All Done! <<", "light_green") + "\n")
                elapsed_time = end_time - start_time
                print(f"程序运行耗时: {elapsed_time} 秒")
                time.sleep(1)
    except Exception as e:
        print("FatalError:", colored(str(e), "red"))
        input("Press Enter to Exit...")


if __name__ == "__main__":
    main()

压缩好之后写了个bat文件直接一键上传/同步

@echo off

rem 正在与cloudflareR2同步
echo Running rclone sync to cf-r2:realxlfd
rclone sync ./Storage cf-r2:resource -P --transfers=50

echo All Storage synced successfully
pause

V2.1?

朋友说需要一个主色调功能，也就是获取随机图的主题色

加上博客为了移动端适配，需要将图片分类为宽图片和窄图片（ratio>1与ratio<1)

于是乎：


from PIL import Image
import concurrent.futures
import os
import threading
import configparser
import time
from colorthief import ColorThief
import json
import logging
import shutil

# 配置日志记录
logging.basicConfig(
    level=logging.INFO,  # 设置日志级别为 INFO
    filename='app.log',  # 指定日志文件路径
    filemode='w',        # 设置文件模式为写入模式
    format='%(asctime)s - %(levelname)s - %(message)s'  # 设置日志格式
)

# 线程锁,防止输出混乱
output_lock = threading.Lock()
# 全局变量
json_data = {}
mainTone_lock = threading.Lock()
count_large_lock = threading.Lock()
count_small_lock = threading.Lock()
log_counter = threading.Lock()
# 默认格式
new_width = 2560  # 新图片宽度
saveQuality = 90  # 输出图片质量：1-100
saveQuality_com = 40  # 输出图片质量：1-100
saveFormat = "webp"  # 输出图片格式
allowed_extensions = [".jpg", ".webp", ".png", ".jpeg"]
workingDir = "./"
num_threads = 10
config = configparser.ConfigParser()
wallpaperDir = "alist/"
mobileDir = "mobile/"
new_width_com = 256
storageDir = "Storage/"

# Count
counter_success_save = 0
counter_success_resize = 0
counter_error = 0
large_img_count = 0
small_img_count = 0


# 读取Config
try:
    config.read("config.ini", encoding="utf-8")
    num_threads = config.getint("FUNCTION", "Threads")
    allowed_extensions = config.get("FILTER", "imgType").split(",")
    workingDir = config.get("FUNCTION", "WorkingDir")
    wallpaperDir = config.get("FUNCTION", "PCWallDir")
    storageDir = config.get("FUNCTION", "StorageDir")
    copyDir = config.get("FUNCTION", "CopyDir")
    mobileDir = config.get("FUNCTION", "MobileWallDir")
    new_width = config.getint("OUTPUT", "ResizeWidth")
    new_width_com = config.getint("OUTPUT", "ResizeWidthCom")
    saveFormat = config.get("OUTPUT", "SaveFormat")
    saveQuality = config.getint("OUTPUT", "SaveQuality")
    saveQuality_com = config.getint("OUTPUT", "SaveQuality_com")
    if workingDir == "":
        workingDir = "./"
    print("\033[1;31m" + "Config Loaded:" + "\033[0m")
    with open("config.ini", "r", encoding="utf-8") as f:
        lines = f.read().splitlines()
        filtered_lines = [line for line in lines if line and not line.startswith("#")]
        text = "\n".join(filtered_lines)
        print("\033[1;32m" + text + "\033[0m")
except Exception as e:
    print("\033[1;31m" + "ConfigLoadError:", e, "\033[0m")
    time.sleep(3)
    exit()

# 预构建输出路径
output_path_largeimg = os.path.join(workingDir, storageDir, "sites/")
output_path_smallimg = os.path.join(workingDir, storageDir, "sites_compressed/")
os.makedirs(output_path_largeimg, exist_ok=True)
os.makedirs(output_path_smallimg, exist_ok=True)


# 获取文件数量
def get_file_count():
    global large_img_count, small_img_count
    scanTarget_largeimgs = os.path.join(workingDir, storageDir, "sites/", wallpaperDir)
    scanTarget_smallimgs = os.path.join(workingDir, storageDir, "sites/", mobileDir)
    os.makedirs(scanTarget_largeimgs, exist_ok=True)
    os.makedirs(scanTarget_smallimgs, exist_ok=True)
    file_list = os.listdir(scanTarget_largeimgs)
    large_img_count = len(file_list)
    file_list = os.listdir(scanTarget_smallimgs)
    small_img_count = len(file_list)


# 判断是否为有效图片
def IsValidImage(img_path):
    bValid = True
    try:
        Image.open(img_path).verify()
    except:
        bValid = False
    return bValid


# 读取json文件
def add_to_json(key, value):
    global json_data
    with mainTone_lock:
        json_data[key] = value


def write_json_to_file():
    global json_data
    try:
        mainTone_path = os.path.join(workingDir, storageDir, "functions/", "mainTone.json")
        os.makedirs(os.path.dirname(mainTone_path), exist_ok=True)
        json_data_temp = {}
        try:
            with open(mainTone_path, "r") as file:
                json_data_temp = json.load(file)
        except:
            pass
        for key, value in json_data.items():
            json_data_temp[key] = json_data[key]
        with open(mainTone_path, "w") as file:
            json.dump(json_data_temp, file)
    except Exception as e:
        logging.error("WriteJsonError:" + str(e))


# 压缩图片(线程)
def compressImage(image_path,image_name):
    global counter_success_resize,counter_success_save, large_img_count, small_img_count
    try:
        saveFormat_temp = saveFormat
        is_compressed = False
        # 预构建输出路径
        output_path = os.path.join(workingDir, "ErrorImage/")
        # 打开图片
        img = Image.open(os.path.join(image_path, image_name))
        # 由宽度判断是否需要压缩
        width, height = img.size[0], img.size[1]
        # 长宽比筛选
        if width / height >= 1.1:
            # 如果为PC壁纸
            final_path = wallpaperDir
            with count_small_lock:
                large_img_count += 1
                image_basename = str(large_img_count)
        else:
            # 如果为手机壁纸
            final_path = mobileDir
            with count_small_lock:
                small_img_count += 1
                image_basename = str(small_img_count)
        # 生成输出路径
        output_filepath = os.path.normpath(os.path.join(
            workingDir,
            storageDir,
            "sites",
            final_path,
            image_basename + "." + saveFormat_temp,
        ))
        output_filepath_com = os.path.join(
            workingDir,
            storageDir,
            "sites_compressed/",
            final_path,
            image_basename + "." + saveFormat_temp,
        )
        os.makedirs(os.path.dirname(output_filepath), exist_ok=True)
        os.makedirs(os.path.dirname(output_filepath_com), exist_ok=True)
        # 生成缩略图
        new_height_com = int(new_width_com * height / width)
        new_img_compressed = img.resize((new_width_com, new_height_com), Image.LANCZOS)
        new_img_compressed.save(
            output_filepath_com, saveFormat_temp, quality=saveQuality_com
        )
        new_img_compressed.close()
        # 判断是否需要压缩
        if width > new_width:
            # 压缩图片
            new_height = int(new_width * height / width)
            new_img = img.resize((new_width, new_height), Image.LANCZOS)
            # 保存图片
            try:
                new_img.save(output_filepath, saveFormat_temp, quality=saveQuality)
                is_compressed = True
            except Exception as e:
                print("Error:", e)
            finally:
                img.close()
                new_img.close()
        else:
            # 不压缩
            try:
                os.rename(os.path.join(image_path, image_name),output_filepath)
                is_compressed = False
            except Exception as e:
                print("Error:", e)
            finally:
                img.close()
        # 提取主色调
        color_thief = ColorThief(output_filepath)
        dominant_color = color_thief.get_color(quality=10)
        hex_color = "#{:02x}{:02x}{:02x}".format(
            dominant_color[0], dominant_color[1], dominant_color[2]
        )
        # 保存主色调: 先将主色调临时变量，再合并到主文件
        json_keyname = '/'+final_path+image_basename
        add_to_json(json_keyname, hex_color)
        with output_lock:
            if is_compressed:
                print("已压缩图片：" + output_filepath, "主色调：" + hex_color)
                with log_counter:
                    counter_success_resize += 1
            print("已保存原始图片：" + output_filepath, "主色调：" + hex_color)
            with log_counter:
                counter_success_save += 1
    except Exception as e:
        with output_lock:
            logging.error("Error:", e)


def process_file(root, file):
    global counter_error
    fileBaseName, extension = os.path.splitext(file)
    extension = extension.lower()
    if extension in allowed_extensions:
        if IsValidImage(os.path.join(root, file)):
            compressImage(root, file)
            if os.path.exists(os.path.join(root, file)):
                os.remove(os.path.join(root, file))
        else:
            relativePath = os.path.relpath(root, workingDir)
            os.makedirs(
                os.path.join(workingDir, "ERROR_IMAGES/", relativePath), exist_ok=True
            )
            os.rename(
                os.path.join(root, file),
                os.path.join(workingDir, "ERROR_IMAGES", relativePath, file),
            )
            with output_lock:
                print("打开失败，已移动无效图片：" + file)
                with log_counter:
                    counter_error += 1
    else:
        with output_lock:
            print(file + "不是图片文件,跳过")

#  清除空文件夹
def remove_empty_folders(folder_path):
    for root, dirs, files in os.walk(folder_path, topdown=False):
        for dir_name in dirs:
            dir_path = os.path.join(root, dir_name)
            if not os.listdir(dir_path):  # 检查文件夹是否为空
                os.rmdir(dir_path)  # 删除空文件夹
                print("删除空文件夹：" + dir_path)

# 同步目录
def sync_dir(target_folder):
    source_folder_pc = os.path.join(workingDir, storageDir, "sites/", wallpaperDir)
    source_folder_mobile = os.path.join(workingDir, storageDir, "sites/", mobileDir)
    target_folder_pc = os.path.join(target_folder, 'PC/')
    target_folder_mobile = os.path.join(target_folder, 'Mobile/')
    if os.path.exists(target_folder_pc):
        shutil.rmtree(target_folder_pc)
    if os.path.exists(target_folder_mobile):
        shutil.rmtree(target_folder_mobile)
    shutil.copytree(source_folder_pc, target_folder_pc)
    shutil.copytree(source_folder_mobile, target_folder_mobile)

# 主函数
def main():
    os.chdir(workingDir)
    os.makedirs(os.path.join(workingDir, "INPUT/"), exist_ok=True)
    get_file_count()
    with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
        for root, dir, filenames in os.walk(os.path.join(workingDir, "INPUT/")):
            for file in filenames:
                executor.submit(process_file, root, file)
    output_string = "处理完成，移动" + str(counter_success_save) + "张，" ,"压缩"+str(counter_success_resize) + "张，"+ "失败" + str(counter_error) + "张"
    print(output_string)
    logging.info(output_string)
    print("正在写入主色调文件...")
    write_json_to_file()
    print("写入完成,正在清理空文件夹...")
    remove_empty_folders(os.path.join(workingDir, "INPUT/"))
    print("清理完成,正在同步目录...")
    sync_dir(copyDir)
    input('按任意键继续...')
    exit()


if __name__ == "__main__":
    main()

api太慢了怎么办？前端缓存

对于用户而言:

1 2	index.html -> xxx.js / xxx.css -> 执行js -> 请求图片 -> 背景加载替换 -> 刷新 -> 请求图片 -> 背景加载替换

电脑是很快的，似乎问题落在了请求图片的身上
但是，cf worker的返回速度就那么快
不如说，所有的后端就那样，因为图片响应速度不仅仅受限于两端的ping，还有用户带宽
有没有一种办法能够让刷图的速度不受api响应速度的影响呢？
似乎service worker解决了这个问题

仔细看看
嗯
这就是我要找的，一个在前端运行的后端，太棒了
开搞

script.js

// SERVICE WORKER
const SW = {
 url: "/service-worker.js?v3",
 register: async () => {
  try {
   if ("serviceWorker" in navigator) {
    await navigator.serviceWorker.register(SW.url);
    await navigator.serviceWorker.ready;
    localStorage.setItem("SWQuery", SW.url);
    if (!navigator.serviceWorker.controller) {
     return;
    }
    const preloadQuery = localStorage.getItem("preloadQuery");
    if (!preloadQuery) {
     SW.addUrls(await SW.getCacheUrls());
     localStorage.setItem("preloadQuery", SW.url);
    }
   }
  } catch {
   console.warn("Unable to register service worker.", err);
  }
 },

 unregister: () => {
  if ("serviceWorker" in navigator) {
   navigator.serviceWorker.getRegistrations().then((registrations) => {
    for (let registration of registrations) {
     registration.unregister();
    }
   });
  }
 },
 // * 获取所需缓存的url
 getCacheUrls: async () => {
  let urls = [];
  // ADD CACHE FOR POST COVERS
  const postImgs = document.querySelectorAll(".post_cover a img");
  postImgs.forEach((postImg) => {
   const targetUrl = postImg.getAttribute("data-lazy-src");
   if (targetUrl) {
    urls.push(targetUrl);
    urls.push(targetUrl + "!imageAve");
   }
  });
  // ICONS CACHE
  const icons = document.querySelectorAll(".back-menu-item img");
  icons.forEach((icon) => {
   const targetUrl = icon.getAttribute("data-lazy-src");
   if (targetUrl) {
    urls.push(targetUrl);
   }
  });
  // ADD CACHE FOR RPICS
  let params = RpicsApi.params;
  const picNum = 4; // * 缓存队列数量
  const rpicsQueue = []; // * 存储rid的队列
  for (let i = 0; i < picNum; i++) {
   const rid = Math.floor(Math.random() * 100000);
   rpicsQueue.push(rid);
   // * 将所有可能的组合都添加至缓存队列
   [params.sizes.THUMBNAIL, params.sizes.PREVIEW].forEach((size) => {
    [params.aspect.PC, params.aspect.MOBILE].forEach((aspect) => {
     urls.push(RpicsApi.getUrl(size, rid, aspect));
    })
   })
  }
  localStorage.setItem("rpicsQueue", JSON.stringify(rpicsQueue));
  // ADD EXTRA RES
  const jsonData = await (await fetch("/sw-cache.json")).json();
  const extraUrls = jsonData.urls;
  for (let url of extraUrls) {
   urls.push(url);
  }
  return urls;
 },
 updateRpicsQueue: (rawQueue) => {
  let arr = [];
  const rpicsQueue = JSON.parse(rawQueue);
  const randomNum = Math.floor(Math.random() * 10000);
  rpicsQueue.push(randomNum);
  arr.push(rpicsQueue.shift());
  arr.push(rpicsQueue[1]);
  localStorage.setItem("rpicsQueue", JSON.stringify(rpicsQueue));
  SW.addUrls(SW.genergateUrls(randomNum));
  return arr;
 },

 addUrls: async (urls) => {
  navigator.serviceWorker.controller.postMessage({
   command: "addCache",
   urls: urls,
  });
 },
 deleteUrls: async (urls) => {
  navigator.serviceWorker.controller.postMessage({
   command: "deleteCache",
   clearUrls: urls,
  });
 },
 updatePjaxUrls: (rid) => {
  const sizes = RpicsApi.params.sizes
  const aspects = RpicsApi.params.aspect
  const [s, ms, l, ml] = RpicsApi.getAllUrlWithRid(rid)
  config.smallSrc = s;
  config.largeSrc = l;
  config.mobileSmallSrc = ms;
  config.mobileLargeSrc = ml;
 },
 genergateUrls: (rid) => {
  const arr = [...RpicsApi.getAllUrlWithRid(rid)]
  return arr
 },
};

// PRE REFRESH
if (!navigator.serviceWorker.controller) {
 SW.register().then(() => {
  location.reload();
 });
}
const SWQuery = localStorage.getItem("SWQuery");
if (SWQuery !== SW.url) {
 caches.delete("WebCache-v1");
 SW.register().then(() => {
  localStorage.removeItem("preloadQuery");
  location.reload();
 });
}
// PROGRESSIVE LOAD
if (document.readyState === "complete") {
 onPJAXComplete();
} else {
 window.addEventListener("load", onPJAXComplete);
}
document.addEventListener("pjax:complete", onPJAXComplete);

service_worker.js

// Cache name: WebCache
// INSTALL
const CACHE_NAME = "WebCache-v1";
this.addEventListener("install", (event) => {
 self.skipWaiting();
 console.warn("Service Worker installed");
});
// TAKE OVER FETCH
this.addEventListener("fetch", (event) => {
 event.respondWith(
  caches.match(event.request).then((response) => {
   if (response) {
    return response;
   }
   SWHandler.cacheTargetUrl(event.request.url);
   return fetch(event.request);
  })
 );
});
// LISTENING ACTIVATE AND REMOVE OLD CACHE
this.addEventListener("activate", (event) => {
 event.waitUntil(
  caches.keys().then((cacheNames) => {
   return Promise.all(
    cacheNames.map((cacheName) => {
     if (cacheName !== CACHE_NAME) {
      return caches.delete(cacheName);
     }
    })
   );
  })
 );
});

// LISTENING MESSAGE
this.addEventListener("message", (event) => {
 if (event.data.command === "addCache") {
  // FILTER DUPLICATE URL
  const fetchUrls = new Set(event.data.urls);
  event.waitUntil(
   caches.open(CACHE_NAME).then((cache) => {
    const urlArr = Array.from(fetchUrls);
    const fetchPromises = urlArr.map((url) => {
     cache.match(url).then((response) => {
      if (!response) {
       // console.log("cacheAdd:" + url);
       return cache.add(url);
      }
     });
    });
    return Promise.all(fetchPromises);
   })
  );
 }
 if (event.data.command === "deleteCache") {
  const deleteUrls = new Set(event.data.clearUrls);
  event.waitUntil(
   caches.open(CACHE_NAME).then((cache) => {
    const urlArr = Array.from(deleteUrls);
    const deletePromises = urlArr.map((url) => {
     cache.match(url).then((response) => {
      if (response) {
       // console.log("delete:" + url);
       return cache.delete(url);
      }
     });
    });
    return Promise.all(deletePromises);
   })
  );
 }
});

const SWHandler = {
 filterHost: ["npm.onmicrosoft.cn", "npm.elemecdn.com"],
 imgCacheHost: ["...", "..."],
 cacheTargetUrl: async (url) => {
  const target = new URL(url);
  if (SWHandler.filterHost.includes(target.hostname)) {
   const cache = await caches.open(CACHE_NAME);
   const response = await cache.match(url);
   if (!response) {
    // console.log("cacheAdd:" + url);
    cache.add(url);
   }
  } else {
   if (SWHandler.imgCacheHost.includes(target.hostname)) {
    if (url.endsWith(".jpg") || url.endsWith(".png") || url.endsWith(".gif") || url.endsWith(".jpeg") || url.endsWith(".webp")|| url.endsWith("!imageAve")) {
     const cache = await caches.open(CACHE_NAME);
     const response = await cache.match(url);
     if (!response) {
      // console.log("cacheAdd:" + url);
      cache.add(url);
     }
    }
   }
  }
 },
};

解决了什么

用户第一次访问网页时注入server worker
缓存所有指定host的请求，以及缓存当前图片后面的4张图片，每次消耗队列中的一张都会新缓存一张，确保队列始终有4张图片缓存
这样用户刷图直接使用缓存而避免发送请求接受响应，网站流畅度大提升，极致丝滑

似乎就这样了，时间定格在了那个暑假结束前夕

明明写了脚本，可是我几乎一次也没更新过那些图片

Refactor

时光飞逝
大概是24年年初
偶然间看到一个WebAssembly图像处理库Photon，觉得这实在是太棒了
想着能不能搬到cloudflare worker上
也就是说：

上传一张图片，只需要post一次，worker接收到数据后会自动压缩转换格式并且放置于正确的位置
岂不美哉？
正好那时候cf d1刚刚公测
开搞！
Github rpics-worker
上传到github上面了
新功能跟下一个版本的差不多，会在后面提到

Fatal Error

完了
完了
cf worker对每个worker有128m内存限制
这意味着什么呢？
假设一张4k的图片需要进行处理，在不支持流式处理的情况下，图片约24M,实际占用为input + output + process cache，也就是至少24M * 3 = 64M,再加上运行时本身的开销
直接给我拒绝服务
咋办？
爆大米
我真的是草了爆了大米买的workers 付费计划还是那金子一样的128M内存
说金子问题是免费也是128M，说免费问题是掏了刀乐还是那128M
遇到大图片直接崩
真的饿饿了

Go V3.0

当时在写golang，想着直接把worker弃了，沟槽的内存限制，幸亏我还有一台hk的小鸡，虽然只有1g内存，跑个golang应该没啥问题

goland，启动

go + gin + sqlite

不错

Github rpics-api-go

新功能

支持指定图片格式，比如说url配置format=webp则将图片转换为webp后返回，如果是第一次请求则会同时缓存转换结果，避免后续的请求再次浪费时间在转换上
支持更细粒度地过滤图片长宽：比如说可以通过>1.3指定返回长宽比大于1.3的图片
提供了去重，即自动忽略用户上传的重复图片（通过hash）
可以通过size指定多个不同尺寸，比如说480p,720p,1080p,2k,4k

功能大概跟workers差不多，但是在自托管的情况下真的快了太多
打包成docker之后运行基本上也不用维护

终点：V4.0?

大学必修课教rust我是万万没想到的
本来就久仰rust大名，打算考完研再去钻研一下，没想到被强迫提前到了现在
抱着试试的心态，随便看了两眼文档
我去
蟹神
不知道为啥，真的一见钟情了——
我感觉rust比js简单多了。。别提python，因为难度排行在我心里是这样的: python>c++/c>js>java>rust>go（个人观点）
那么简单，性能又那么好
为啥不用？

但是，api结构再怎么优化就那样
到底能改什么呢？

immich ，一个自托管相册库
无意中在unraid 应用中心刷到
大概在5月份？就把自己的涩图全部转移到immich相册里去了

一个点子逐渐浮现：如果…我是说如果
能把随机图api建立在immich上层呢？
以immich相册的形式组织图片，而随机图api的数据是immich相册的缓存…
我去
这样难加图的问题，难维护的问题全部迎刃而解了？
immich对于每一张图片也存储了3个版本（thumbnail/preview/original)，并且提供了去重
这样图片尺寸转换、格式转换、去重不都解决了？
官方还提供了详细的api说明文档
简直就是专业对口

于是我赶紧打开rustover
呸
vscode
为什么不用rustover？
因为analyzer的速度太几把慢了，我敲个代码得等3-4s才有提示，何况我是尊贵的非13/14代无锁intel i7
vscode，启动！

找web框架，看到rocket的文档很全面喜欢，用了。
找orm框架，我去看不懂一点，稍微试了下也报错
哎
还是直接写sql吧

于是，确定下来rust + rocket + sqlx

开搞

你以为我要放仓库连接？哈哈我自己这个版本都被我删了
为啥？
因为我™用rocket、axum和actix都写了一遍…

V4.0

请求->数据库->文件->流式响应
请求->数据不存在->请求immich->流式响应+写入文件
emm
大概就这样
随便写了个初版
一压测
1w8到顶了(12700k)，sqlite同时疯狂报错提示响应时间过长
哈哈
受不了
加个内存索引和内存缓存吧
请求->缓存->响应
请求->缓存（未命中）->数据库->文件->响应
…
emm
ok加个预缓存
ok加个自动索引刷新

开测

我的12700k上跑到6w6 qps到顶了，此时cpu占用大概30%左右，内存150MB
连接数不记得了，好像是1000？

我还以为能上10w
有点垃

perf 测个火焰图，handler只占了25%左右，大部分消耗全在rocket framework上，还有醒目的hyper
得换框架了…

V4.1

Mutex<HashMap<String,...>>
有点垃
换成DashMap<FastStr,...>
rocket有点垃，换成axum试试?
……
写了出来
我去axum也是hyper
rocket和axum都是hyper
哎真的别hyper了
于是用了actix-web
windows上一测，发现…
3300qps?
真奇怪，至少也得3w吧？
但我打开任务管理器，发现4个大字：“效能模式”
哈哈^_^
诶
不是哥们我压测怎么cpu占用率只有0.5%

有点无敌了
没搞懂
于是ssh连接我的12300 + unraid开的 debian + 8gb内存的虚拟机

1	cargo build --release

wrk，启动！
不是
19w qps了直接
1s传4.5gb数据
逆天了
这就是蟹神吗？
10000连接数也轻轻松松16w qps
Platform
Wrk
于是，就有了现在的版本，也就是这个博客正在用的版本
V4.1.x

以后遇到好的灵感我也会继续更新重构的喵
谢谢你读到这里，哈哈

不放仓库，因为我是彩笔，怕有人提issue骂我

我github都不会用真的对不起了

想开源就在下面骂我，谢谢

不过我知道文章101%没人看，所以是写给自己的，嘻嘻喵