chore: Replace deprecated resolve conflicts field

Error: creating EKS Add-On (wandb-qa-aws:vpc-cni): ResourceInUseException: Addon already exists. { RespMetadata: { StatusCode: 409, RequestID: "c38e4e71-d773-42f6-8f27-164471b732bf" }, Message_: "Addon already exists." }

This PR is included in version 4.12.1 :tada:


pd doesn't behave well under heavy sync load

Today on Testnet 56 we observed a large spike in client traffic to the pd endpoint at https://grpc.testnet.penumbra.zone.

As for provenance of the traffic, let's assume it's organic interest, in the form of many people downloading the web extension and synchronizing blocks for the first time. After about an hour or two, memory consumption—in the pd container specifically—balloons to the point that OOMkiller kicks in and terminates the pod. An example of resource consumption shortly before kill:
❯ kubectl top pod penumbra-testnet-fn-0-54d9f68d76-kl9gm --containers
POD                                      NAME   CPU(cores)   MEMORY(bytes)
penumbra-testnet-fn-0-54d9f68d76-kl9gm   pd     3055m        11249Mi
penumbra-testnet-fn-0-54d9f68d76-kl9gm   tm     47m          162Mi

According to the logs, pd is serving a lot of two types of requests, CompactBlockRange and ValidatorInfoStream:
❯ wc -l logs-penumbra-testnet-fn-0-54d9f68d76-kl9gm.txt
8329 logs-penumbra-testnet-fn-0-54d9f68d76-kl9gm.txt
❯ rg validator_info logs-penumbra-testnet-fn-0-54d9f68d76-kl9gm.txt -c
4639
❯ rg -i compact_block_range logs-penumbra-testnet-fn-0-54d9f68d76-kl9gm.txt -c
2107

Intriguingly both those types are Boxed return values in our RPCs. Also intriguing is this comment:
https://github.com/penumbra-zone/penumbra/blob/bfda3a85ba92b1d4112ebe402c0fb2a8e6271c60/crates/bin/pd/src/info/oblivious.rs#L332-L336
We need to understand why pd consumes large amounts of memory when handling these types of concurrent requests. For now, I'm assuming the traffic is well-formed, honest clients trying to synchronize.

Maybe https://github.com/koute/bytehound would be useful here? I think a good goal would be that pd can maintain many active connections without using excess memory.

I don't think that the use of an explicit Box is a meaningful indicator of what might be consuming significant per-connection memory.  We should investigate using a heap profiler.
On the assumption that each connection is using the same amount of memory (a reasonable prior, though it could certainly not be the case), it might be sufficient to collect a heap profile of what happens when a single client connection is held open. If there's significant per-connection memory overhead, that's a bug, and it could show up without having to load many connections.

It seems we do have a memory leak in pd. I can fairly easily get pd to consume a few gigs of memory if I bombard it with sync requests from multiple clients. Using bytehound to profile as recommended above, we see some never-freed allocations:

We can also show that bytehound believes these are leaks:

Unfortunately I don't yet have a root cause, but will spend more time with the stack traces and try to piece together a clearer story. The bytehound guide gives a walkthrough about how to perform this kind of investigation. I pushed the testing script I used (very simple loop over pcli operations) for visibility.

Nice digging!
Not sure if it will be helpful, but one thought about isolating a cause could be:

run pd + tendermint
shut off tendermint so pd is not processing any consensus messages or doing anything other than serving rpc requests
sync pclientd against the stalled pd until chain tip, then turn it off
start measuring pd
start pclientd, causing it to open a single long-lived connection
stop measuring pd

This way, we might be able to get information about what memory is used by a single long-lived connection.

This way, we might be able to get information about what memory is used by a single long-lived connection.

Ahoy, thar she blows:

Reading through the stack trace associated with that leak, I see a lot of rocksdb references, so I'm guessing that we're not dropping a db handle in a service worker somewhere.

full stack trace for graphed leak

#00 [libc.so.6] __clone3
#01 [libc.so.6] start_thread
#02 [libstdc++.so.6.0.30] 7f57e02dbc42
#03 [pd] std::_Function_handler<void (), rocksdb::VersionBuilder::Rep::LoadTableHandlers(rocksdb::InternalStats*, int, bool, bool, std::shared_ptr<rocksdb::SliceTransform const> const&, unsigned long)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
#04 [pd] rocksdb::TableCache::FindTable(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::BasicTypedCacheInterface<rocksdb::TableReader, (rocksdb::CacheEntryRole)13, rocksdb::Cache*>::TypedHandle**, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, rocksdb::HistogramImpl*, bool, int, bool, unsigned long, rocksdb::Temperature)
#05 [pd] rocksdb::TableCache::GetTableReader(rocksdb::ReadOptions const&, rocksdb::FileOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, bool, bool, rocksdb::HistogramImpl*, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader>>*, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, int, bool, unsigned long, rocksdb::Temperature)
#06 [pd] rocksdb::BlockBasedTableFactory::NewTableReader(rocksdb::ReadOptions const&, rocksdb::TableReaderOptions const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader>>&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader>>*, bool) const [clone .localalias]
#07 [pd] rocksdb::BlockBasedTable::Open(rocksdb::ReadOptions const&, rocksdb::ImmutableOptions const&, rocksdb::EnvOptions const&, rocksdb::BlockBasedTableOptions const&, rocksdb::InternalKeyComparator const&, std::unique_ptr<rocksdb::RandomAccessFileReader, std::default_delete<rocksdb::RandomAccessFileReader>>&&, unsigned long, std::unique_ptr<rocksdb::TableReader, std::default_delete<rocksdb::TableReader>>*, std::shared_ptr<rocksdb::CacheReservationManager>, std::shared_ptr<rocksdb::SliceTransform const> const&, bool, bool, int, bool, unsigned long, bool, rocksdb::TailPrefetchStats*, rocksdb::BlockCacheTracer*, unsigned long, std::string const&, unsigned long, std::array<unsigned long, (unsigned long)2>)
#08 [pd] rocksdb::BlockBasedTable::PrefetchIndexAndFilterBlocks(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, rocksdb::BlockBasedTable*, bool, rocksdb::BlockBasedTableOptions const&, int, unsigned long, unsigned long, rocksdb::BlockCacheLookupContext*)
#09 [pd] rocksdb::BlockBasedTable::CreateIndexReader(rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader>>*)
#10 [pd] rocksdb::BinarySearchIndexReader::Create(rocksdb::BlockBasedTable const*, rocksdb::ReadOptions const&, rocksdb::FilePrefetchBuffer*, bool, bool, bool, rocksdb::BlockCacheLookupContext*, std::unique_ptr<rocksdb::BlockBasedTable::IndexReader, std::default_delete<rocksdb::BlockBasedTable::IndexReader>>*)
#11 [pd] rocksdb::BlockBasedTable::IndexReaderCommon::ReadIndexBlock(rocksdb::BlockBasedTable const*, rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, bool, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, rocksdb::CachableEntry<rocksdb::Block>*)
#12 [pd] std::enable_if<((rocksdb::Block_kIndex::kCacheEntryRole)==((rocksdb::CacheEntryRole)13))||(true), rocksdb::Status>::type rocksdb::BlockBasedTable::RetrieveBlock<rocksdb::Block_kIndex>(rocksdb::FilePrefetchBuffer*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::UncompressionDict const&, rocksdb::CachableEntry<rocksdb::Block_kIndex>*, rocksdb::GetContext*, rocksdb::BlockCacheLookupContext*, bool, bool, bool) const
#13 [pd] rocksdb::Status rocksdb::(anonymous namespace)::ReadAndParseBlockFromFile<rocksdb::Block_kIndex>(rocksdb::RandomAccessFileReader*, rocksdb::FilePrefetchBuffer*, rocksdb::Footer const&, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, std::unique_ptr<rocksdb::Block_kIndex, std::default_delete<rocksdb::Block_kIndex>>*, rocksdb::ImmutableOptions const&, rocksdb::BlockCreateContext&, bool, rocksdb::UncompressionDict const&, rocksdb::PersistentCacheOptions const&, rocksdb::MemoryAllocator*, bool, bool)
#14 [pd] rocksdb::BlockFetcher::ReadBlockContents()
#15 [pd] rocksdb::UncompressSerializedBlock(rocksdb::UncompressionInfo const&, char const*, unsigned long, rocksdb::BlockContents*, unsigned int, rocksdb::ImmutableOptions const&, rocksdb::MemoryAllocator*)
#16 [pd] rocksdb::UncompressBlockData(rocksdb::UncompressionInfo const&, char const*, unsigned long, rocksdb::BlockContents*, unsigned int, rocksdb::ImmutableOptions const&, rocksdb::MemoryAllocator*)
#17 [libstdc++.so.6.0.30] operator new(unsigned long)
#18 [libbytehound.so] malloc [api.rs:294]

I also saw a smaller leak that may be related to tracing instrumentation, or else I'm mistaken in reading the backtrace. Here's a PDF of the full report I generated this morning, mostly for posterity in reproducing these steps in future debugging sessions:
pd-memory-profiling-testnet-56-report-1.pdf
Formatting's a bit wonky. Separately I'll paste in the console code I adapted from the bytehound guide, since that'll be fairly easily copy/pasteable in the future.

bytehound memleak console scripting

println("Temporary vs leaked allocations:");
graph()
    .add("Leaked", allocations().only_leaked())
    .add("Temporary", allocations())
    .save();

println("Sort leaks by size:");
let groups = allocations()
    .only_leaked()
    .group_by_backtrace()
        .sort_by_size();
graph().add(groups).save();

// Function to group leaks based on common backtraces
fn analyze_group(list) {
    let list_all = allocations().only_matching_backtraces(list);
    graph()
        .add("Leaked", list_all.only_leaked())
        .add("Temporary", list_all)
        .save();
    println("Total: {}", list_all.len());
    println("Leaked: {}", list_all.only_leaked().len());
    println();
    println("Backtrace:");
    println(list_all[0].backtrace().strip());
}

println("Analyzing most common leak by backtrace:");
analyze_group(groups[0]);
println("Analyzing second most common leak by backtrace:");
analyze_group(groups[1]);
println("Analyzing third most common leak by backtrace:");
analyze_group(groups[2]);

Initial drops in linked PR seem to help, but aren't sufficient. Encountered a new leak. Stack trace:
#00 [libc.so.6] __clone3
#01 [libc.so.6] start_thread
#02 [pd] std::sys::unix::thread::Thread::new::thread_start [thread.rs:108]
#03 [pd] <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once [boxed.rs:1985]
#04 [pd] <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once [boxed.rs:1985]
#05 [pd] core::ops::function::FnOnce::call_once{{vtable.shim}}
#07 [pd] tokio::runtime::blocking::pool::Inner::run
#08 [pd] tokio::runtime::task::harness::Harness<T,S>::poll
#09 [pd] tokio::runtime::task::core::Core<T,S>::poll
#10 [pd] tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
#11 [pd] <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
#12 [pd] tokio::runtime::scheduler::multi_thread::worker::run
#13 [pd] tokio::runtime::context::runtime::enter_runtime
#14 [pd] tokio::runtime::context::scoped::Scoped<T>::set
#15 [pd] tokio::runtime::scheduler::multi_thread::worker::Context::run
#16 [pd] tokio::runtime::scheduler::multi_thread::worker::Context::run_task
#17 [pd] tokio::runtime::task::harness::Harness<T,S>::poll
#18 [pd] tokio::runtime::task::core::Core<T,S>::poll
#19 [pd] tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
#20 [pd] <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
#21 [pd] <hyper::proto::h2::server::H2Stream<F,B> as core::future::future::Future>::poll
#22 [pd] <hyper::proto::h2::PipeToSendStream<S> as core::future::future::Future>::poll
#23 [pd] <http_body::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_data
#24 [pd] <http_body::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_data
#25 [pd] <http_body::combinators::map_err::MapErr<B,F> as http_body::Body>::poll_data
#26 [pd] <tonic::codec::encode::EncodeBody<S> as http_body::Body>::poll_data
#27 [pd] <T as futures_util::fns::FnMut1<A>>::call_mut
#28 [pd] prost::message::Message::encode
#29 [pd] <penumbra_proto::penumbra::core::chain::v1alpha1::CompactBlock as prost::message::Message>::encode_raw
#30 [pd] prost::encoding::message::encode
#31 [pd] prost::encoding::<impl prost::encoding::sealed::BytesAdapter for alloc::vec::Vec<u8>>::append_to
#32 [pd] bytes::bytes_mut::BytesMut::reserve_inner
#33 [pd] alloc::raw_vec::RawVec<T,A>::reserve::do_reserve_and_handle
#34 [pd] alloc::raw_vec::finish_grow
#35 [libbytehound.so] realloc [api.rs:378]

Currently pairing with @erwanor.

The second leak is more mysterious to me, since the allocation is happening inside the Tonic stack, and I'm not sure why it would be growing a BytesMut and then keeping it around. I think it's worth merging a fix for the part of the problem that's clearly in our part of the stack.

Recent related changes:

https://github.com/penumbra-zone/penumbra/pull/2868
https://github.com/penumbra-zone/penumbra/pull/2887
https://github.com/penumbra-zone/penumbra/pull/2895
https://github.com/penumbra-zone/penumbra/pull/2885

We shipped point releases as 0.56.1 and 0.57.1 to evaluate performance improvements. At least one more PR should land in time for 0.58.0 (#2888).

Moving this issue back to Future since we are prioritizing other items. Over the medium term, we should aim for pd to completely isolate the consensus module from inbound load on its RPCs. Although the current performance pattern is far from optimal, it is probably sufficient when combined with network level rate limiters and load balancers.

Closing as completed since we addressed the memory leaks that were causing the original problem. While there is more work to do, it can be tracked in later issues.


Using watermark and rotation during the same request

When I'm using watermark filter, rotation is set to 0. Is it normal or is it any walkaround?

Can you give code examples? How do you apply filters? Please be detailed, so we can reproduce.


Make occurence highlighting delay (occurence_timer) user-configurable

From<EMAIL_ADDRESS>on 2011-05-24T07:18:56Z
Occurence highlighting delay is hardcoded at 1500ms.
Could this be user-configurable in spyder.ini?
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=663

From ccordoba12 on 2011-05-27T08:57:41Z
It's not a bad idea. Thanks.
Summary: Make occurence highlighting delay (occurence_timer) user-configurable
Labels: -Type-Defect Type-Enhancement Cat-Editor

From pierre.raybaut on 2011-05-27T14:49:38Z
This issue was closed by revision 9f2fe9876d8b .

From pierre.raybaut on 2011-05-27T14:49:37Z
This issue was closed by revision 94651a0b555b .
Status: Fixed


LoRa微调训练报错

非常抱歉占用博主您宝贵的时间，麻烦请你帮我解答两个问题

我按照训练微调提示要求在微软商店下载了WSL及Ubuntu，在xu训练的时候报错提示如下

可以不在Windows上安装Ubuntu吗？如果用VM安装Ubuntu可以用吗？
还有这个训练数据集的Prompt格式是哪种呢？

你在windows控制台上输入一下wsl, 看提示的错误是什么, 通过网络搜索解决, 一般是需要安装特定组件或者开启虚拟化功能

windows必须用WSL安装Ubuntu, 因为WSL支持显卡直通

如果你要训练小说, 那么就是整本小说的文本作为一行, 就像你截图的第一行; 如果要训练对话, 则是像第三行那样, 写成对话格式; 如果要训练指令或者代码, 就像第四第五行那样.
总而言之, 你想要lora微调过的模型怎样被使用, 就提供怎样形式的数据.
此外, 建议塞一些与你的微调目标无关的, 各种其他形式的数据(例如对话, 续写), 避免模型过拟合降智

{
"Instruction":   "question ",
"Input":   " background knowledge",
"Response":   "answer",
}
这种方式可以嘛

请问怎么看训练的进度和结果呢？

{ "Instruction": "question ", "Input": " background knowledge", "Response": "answer", }
这种方式可以嘛

不能用这种方式, 看第四行, 基底模型只有这种形式的指令支持, 用你这个形式做lora微调效果不会很好, 建议要么全量微调, 要么参考第四行这种格式
你上面的截图是在安装训练所需的依赖, 安装完毕后开始训练会显示进度的

一般训练大概要多久，仅仅只是测试训练的话，用的0.1B的模型

卡这里不动了，是还在加载中嘛？还是以及暂停了，每太看懂他这执行的意思

再次点击训练, 看是否出现了gcc installed; requirements satisfied

Building dependency tree...
Reading state information...
Package gcc is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
gcc-11-doc gcc-9-doc gcc-12-doc gcc-10-doc
E: Package 'gcc' has no installation candidate
pip installed
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package ninja-build
--2024-01-12 13:59:55-- https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... <IP_ADDRESS>
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://developer.download.nvidia.cn/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin [following]
--2024-01-12 14:00:11-- https://developer.download.nvidia.cn/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
Resolving developer.download.nvidia.cn (developer.download.nvidia.cn)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to developer.download.nvidia.cn (developer.download.nvidia.cn)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 190 [application/octet-stream]
Saving to: ‘cuda-wsl-ubuntu.pin’
0K 100% 26.6M=0s
2024-01-12 14:00:12 (26.6 MB/s) - ‘cuda-wsl-ubuntu.pin’ saved [190/190]
--2024-01-12 14:00:12-- https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... <IP_ADDRESS>
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://developer.download.nvidia.cn/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb [following]
--2024-01-12 14:00:16-- https://developer.download.nvidia.cn/compute/cuda/12.2.0/local_installers/cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb
Resolving developer.download.nvidia.cn (developer.download.nvidia.cn)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to developer.download.nvidia.cn (developer.download.nvidia.cn)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File ‘cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb’ not modified on server. Omitting download.
(Reading database ... 25419 files and directories currently installed.)
Preparing to unpack cuda-repo-wsl-ubuntu-12-2-local_12.2.0-1_amd64.deb ...
Unpacking cuda-repo-wsl-ubuntu-12-2-local (12.2.0-1) over (12.2.0-1) ...
Setting up cuda-repo-wsl-ubuntu-12-2-local (12.2.0-1) ...
Reading package lists...
E: Could not get lock /var/lib/apt/lists/lock. It is held by process 979 (apt-get)
E: Unable to lock directory /var/lib/apt/lists/
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package cuda
requirements satisfied
loading models/RWKV-5-World-1B5-v2-20231025-ctx4096.pth
v5/train.py --vocab_size 65536 --n_layer 24 --n_embd 2048
INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw37_4auo
INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw37_4auo/_remote_module_non_scriptable.py
INFO:pytorch_lightning.utilities.rank_zero:########## work in progress ##########
[2024-01-12 14:01:48,105] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO:pytorch_lightning.utilities.rank_zero:
############################################################################

RWKV-5 BF16 on 1x1 GPU, bsz 1x1x1=1, deepspeed_stage_2 with grad_cp

Data = ./finetune/json2binidx_tool/data/test2_text_document (binidx), ProjDir = lora-models

Epoch = 0 to 19, save every 1 epoch

Each "epoch" = 200 steps, 200 samples, 204800 tokens

Model = 24 n_layer, 2048 n_embd, 1024 ctx_len

Adam = lr 5e-05 to 5e-05, warmup 0 steps, beta (0.9, 0.999), eps 1e-08

Found torch 1.13.1+cu117, recommend 1.13.1+cu117 or newer
Found deepspeed 0.11.2, recommend 0.7.0 (faster than newer versions)
Found pytorch_lightning 1.9.5, recommend 1.9.5

############################################################################
INFO:pytorch_lightning.utilities.rank_zero:{'load_model': 'models/RWKV-5-World-1B5-v2-20231025-ctx4096.pth', 'wandb': '', 'proj_dir': 'lora-models', 'random_seed': -1, 'data_file': './finetune/json2binidx_tool/data/test2_text_document', 'data_type': 'binidx', 'vocab_size': 65536, 'ctx_len': 1024, 'epoch_steps': 200, 'epoch_count': 20, 'epoch_begin': 0, 'epoch_save': 1, 'micro_bsz': 1, 'n_layer': 24, 'n_embd': 2048, 'dim_att': 2048, 'dim_ffn': 7168, 'pre_ffn': 1, 'head_qk': 1, 'tiny_att_dim': 0, 'tiny_att_layer': -999, 'lr_init': 5e-05, 'lr_final': 5e-05, 'warmup_steps': 0, 'beta1': 0.9, 'beta2': 0.999, 'adam_eps': 1e-08, 'grad_cp': 1, 'dropout': 0, 'weight_decay': 0, 'weight_decay_final': -1, 'my_pile_version': 1, 'my_pile_stage': 0, 'my_pile_shift': -1, 'my_pile_edecay': 0, 'layerwise_lr': 1, 'ds_bucket_mb': 200, 'my_sample_len': 0, 'my_ffn_shift': 1, 'my_att_shift': 1, 'head_size_a': 64, 'head_size_divisor': 8, 'my_pos_emb': 0, 'load_partial': 0, 'magic_prime': 0, 'my_qa_mask': 0, 'my_random_steps': 0, 'my_testing': '', 'my_exit': 99999999, 'my_exit_tokens': 0, 'emb': False, 'lora': True, 'lora_load': '', 'lora_r': 8, 'lora_alpha': 32.0, 'lora_dropout': 0.01, 'lora_parts': 'att,ffn,time,ln', 'logger': False, 'enable_checkpointing': False, 'default_root_dir': None, 'gradient_clip_val': 1.0, 'gradient_clip_algorithm': None, 'num_nodes': 1, 'num_processes': None, 'devices': '1', 'gpus': None, 'auto_select_gpus': None, 'tpu_cores': None, 'ipus': None, 'enable_progress_bar': True, 'overfit_batches': 0.0, 'track_grad_norm': -1, 'check_val_every_n_epoch':<PHONE_NUMBER>00000000000, 'fast_dev_run': False, 'accumulate_grad_batches': 8, 'max_epochs': 20, 'min_epochs': None, 'max_steps': -1, 'min_steps': None, 'max_time': None, 'limit_train_batches': None, 'limit_val_batches': None, 'limit_test_batches': None, 'limit_predict_batches': None, 'val_check_interval': None, 'log_every_n_steps':<PHONE_NUMBER>00000000000, 'accelerator': 'gpu', 'strategy': 'deepspeed_stage_2', 'sync_batchnorm': False, 'precision': 'bf16', 'enable_model_summary': True, 'num_sanity_val_steps': 0, 'resume_from_checkpoint': None, 'profiler': None, 'benchmark': None, 'reload_dataloaders_every_n_epochs': 0, 'auto_lr_find': False, 'replace_sampler_ddp': False, 'detect_anomaly': False, 'auto_scale_batch_size': False, 'plugins': None, 'amp_backend': None, 'amp_level': None, 'move_metrics_to_cpu': False, 'multiple_trainloader_mode': 'max_size_cycle', 'inference_mode': True, 'my_timestamp': '2024-01-12-14-01-50', 'betas': (0.9, 0.999), 'real_bsz': 1, 'run_name': '65536 ctx1024 L24 D2048'}
Using /root/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
RWKV_MY_TESTING
Traceback (most recent call last):
File "/mnt/d/LS/Rwkv/./finetune/lora/v5/train.py", line 308, in
from src.trainer import train_callback, generate_init_weight
File "/mnt/d/LS/Rwkv/finetune/lora/v5/src/trainer.py", line 6, in
from .model import LORA_CONFIG
File "/mnt/d/LS/Rwkv/finetune/lora/v5/src/model.py", line 56, in
wkv5_cuda = load(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1284, in load
return _jit_compile(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
_write_ninja_file_and_build_library(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 1597, in _write_ninja_file_and_build_library
get_compiler_abi_compatibility_and_version(compiler)
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 336, in get_compiler_abi_compatibility_and_version
if not check_compiler_ok_for_platform(compiler):
File "/usr/local/lib/python3.10/dist-packages/torch/utils/cpp_extension.py", line 290, in check_compiler_ok_for_platform
which = subprocess.check_output(['which', compiler], stderr=subprocess.STDOUT)
File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['which', 'c++']' returned non-zero exit status 1.
没有出现 gcc installed; requirements satisfied

再次点击训练, 看是否出现了gcc installed; requirements satisfied

有几个依赖在请求的时候为443，是不是这个原因？需要借助梯子嘛？

WSL执行一下 sudo apt update

LoRA additionally training parameter time_mix_r
LoRA additionally training module blocks.8.ffn.key
LoRA additionally training module blocks.8.ffn.receptance
INFO:pytorch_lightning.utilities.rank_zero:########## Loading models/RWKV-5-World-0.1B-v1-20230803-ctx4096.pth... ##########
LoRA additionally training module blocks.8.ffn.value
LoRA additionally training module blocks.9.ln1
LoRA additionally training module blocks.9.ln2
LoRA additionally training parameter time_mix_k
LoRA additionally training parameter time_mix_v
LoRA additionally training parameter time_mix_r
LoRA additionally training parameter time_mix_g
LoRA additionally training parameter time_decay
LoRA additionally training parameter time_faaaa
LoRA additionally training module blocks.9.att.receptance
LoRA additionally training module blocks.9.att.key
LoRA additionally training module blocks.9.att.value
LoRA additionally training module blocks.9.att.gate
LoRA additionally training module blocks.9.att.ln_x
LoRA additionally training parameter time_mix_k
LoRA additionally training parameter time_mix_r
LoRA additionally training module blocks.9.ffn.key
LoRA additionally training module blocks.9.ffn.receptance
LoRA additionally training module blocks.9.ffn.value
LoRA additionally training module blocks.10.ln1
LoRA additionally training module blocks.10.ln2
LoRA additionally training parameter time_mix_k
LoRA additionally training parameter time_mix_v
LoRA additionally training parameter time_mix_r
LoRA additionally training parameter time_mix_g
LoRA additionally training parameter time_decay
LoRA additionally training parameter time_faaaa
LoRA additionally training module blocks.10.att.receptance
LoRA additionally training module blocks.10.att.key
LoRA additionally training module blocks.10.att.value
LoRA additionally training module blocks.10.att.gate
LoRA additionally training module blocks.10.att.ln_x
LoRA additionally training parameter time_mix_k
LoRA additionally training parameter time_mix_r
LoRA additionally training module blocks.10.ffn.key
LoRA additionally training module blocks.10.ffn.receptance
LoRA additionally training module blocks.10.ffn.value
LoRA additionally training module blocks.11.ln1
LoRA additionally training module blocks.11.ln2
LoRA additionally training parameter time_mix_k
LoRA additionally training parameter time_mix_v
LoRA additionally training parameter time_mix_r
LoRA additionally training parameter time_mix_g
LoRA additionally training parameter time_decay
LoRA additionally training parameter time_faaaa
LoRA additionally training module blocks.11.att.receptance
LoRA additionally training module blocks.11.att.key
LoRA additionally training module blocks.11.att.value
LoRA additionally training module blocks.11.att.gate
LoRA additionally training module blocks.11.att.ln_x
LoRA additionally training parameter time_mix_k
LoRA additionally training parameter time_mix_r
LoRA additionally training module blocks.11.ffn.key
LoRA additionally training module blocks.11.ffn.receptance
LoRA additionally training module blocks.11.ffn.value
Traceback (most recent call last):
File "/mnt/d/LS/Rwkv/./finetune/lora/v5/train.py", line 379, in
model.load_state_dict(load_dict, strict=(not args.lora))
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for RWKV:
size mismatch for blocks.0.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.0.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.1.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.1.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.1.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.2.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.2.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.2.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.3.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.3.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.3.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.4.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.4.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.4.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.5.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.5.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.5.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.6.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.6.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.6.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.7.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.7.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.7.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.8.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.8.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.8.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.9.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.9.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.9.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.10.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.10.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.10.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
size mismatch for blocks.11.att.time_decay: copying a param with shape torch.Size([12]) from checkpoint, the shape in current model is torch.Size([12, 64]).
size mismatch for blocks.11.ffn.key.weight: copying a param with shape torch.Size([3072, 768]) from checkpoint, the shape in current model is torch.Size([2688, 768]).
size mismatch for blocks.11.ffn.value.weight: copying a param with shape torch.Size([768, 3072]) from checkpoint, the shape in current model is torch.Size([768, 2688]).
按照你的做了时候重新试了一下，但是又卡了，卡在这里半个小时没动静了，这是报错了还是怎么回事呢

试试RWKV5-1.5B, RWKV5的小尺寸模型版本可能lora没有适配, RWKV5有几个小版本的


Secret in Angular

Hey
I see that you store the secret (acmesecret) in auth.service.ts, isn't this dangerous?
Thanks in advance.

I think it's secure.
This secret only identify your application. Nothing more.
You can't authenticate user with secret but without username and password
You can't sing jwt tokens with secret but without key stored at server side.

Thank you for your answer, oAuth 2 is kinda new for me and really complicated :p

@kucharzyk What happens if I create a malicious website, lets say 'spring-angular2-starter-bad'. I can just decompile the secret from your javascript on the real site, ask for the user to login using their credentials and now I have the token, correct?

@Jdruwe  You have right but it works only when I pass my credentials on your site.
Token has expiration date. What you really need is my credentials.
Suppose you created facebook-bad.com

I visit it and pass my facebook credentials
Now you have my username and password
Do you need token? You have everything you need.

Password grant type is created for trusted clients only.
When you authorize on my page you will be not authorized on your bad page.
If you pass your credentials on malicious site - it's your fault  - it's the same problem like false bank websites

Thank! This kinda summarises it:

After removing your client-secret, I don't think your app is really any safer but it feels a bit cleaner and honest--in that you're acknowledging that using a JS-only client, you can only be so safe. For JS and native clients, the implicit flow is typically used and it doesn't bother with a client-secret. It's simplified from the more robust authorization code grant flow for the fact that with a JS or native client can't keep a secret.


Update Ui to display field name for clarity

This PR will update the Ui as shown below

Closes #96

yes, thank you for pointing that out! deleted in 0aa38ed


StickyExecution: clear stickiness

Most work are on server side: https://github.com/uber/cadence/issues/372
Minor changes are needed on client lib as well.

this is fixed


Revert "fix: #377 window.tauri is undefined upon reloading"

Reverts tauri-apps/tauri#453

Accidentally merged this PR when attempting to merge 452.


PortalWrapper: Fix integration with Route usage

PortalWrapper: Fix integration with Route usage

This commit resolves the issue where Route (react-router) mounted
components built with PortalWrapper (specifically Modals) don't
correctly close when the ESCAPE key is pressed.
This is due to react-router injecting the Modals in a non-linear
order. To resolve this, I've adjusted how PortalWrapper works with
it's GlobalManager. Instead of using IDs, it will use unique
incrementing index number values. This ensures that the latest
injected component, always has the highest index value. Therefore,
when triggere a close (with the escape key), the GlobalManager will
look for the highest index value and execute the close action it.
Unit tests were enhanced, but more importantly, acceptance tests
were added to test out these interactions, especially with
route changes/ESCAPE interactions.
Note: Storybook was updated to the latest versions in this update.
Resolves: https://github.com/helpscout/blue/issues/127

Changes Unknown when pulling 4e38aab7b5ead9c478c785e29933e9312abf5641 on Portal/fix-route-integration into ** on master**.


Dynamically change url of shareon componentswhen prompted

In some cases there are cases in which the location.href is not meant to be the shareable url.

You can use the data-url attribute on either the button or the button container (the one with class shareon). See the documentation here

I have tried this. It seems to only change the url on onload. When the attribute is changed after element is loaded shareon will not catch on to it.

Oh, yes, you have to call Shareon.init() after every update. For some reason, it's not in the documentation... I'll have to adjust that


Add gitter badge to readme (first-timers-only)

first-timers-only
This issue is tagged :octocat: first-timers-only. It is only for people who have never contributed to open source before and are looking for an easy way take their first steps.
Consider this as your chance to dip your toe into the world of open-source and get some bragging rights for writing code that makes drones fly, cars find charging stations, helps people and goods move from place to place and more...
Find more first-timers-only issues from DAV Foundation here.
Thank you for your help :heart:
What is this project?
DAV (Decentralized Autonomous Vehicles) is a new non-profit foundation working to build an open-source infrastructure for autonomous vehicles (cars, drones, trucks, robots, and all the service providers around them) to communicate and transact with each other over blockchain.
As an organization that believes in building a large community of open-source contributors, we often create issues like this one to help people take their first few steps into the world of open source.
n3m0

n3m0 is an autonomous boat being built as a collaboration between the California Maritime University, and the DAV Foundation. This project contains the code running onboard the boat's Raspberry Pi payload computer.
How you can help
Your task for this issue is...
In the README.md file add the following line of code
[![Gitter](https://img.shields.io/gitter/room/DAVFoundation/DAV-Contributors.svg?style=flat-square)](https://gitter.im/DAVFoundation/DAV-Contributors)

This will display a badge like this
Have a look at the readme.md pages of other DAV projects for location of the badge.
Contributing to n3m0

[ ] Make sure this issue is labeled up-for-grabs and not labeled claimed, to verify no one else is working on it.
[ ] Comment in this issue that you would like to do it.
[ ] Open the n3m0 GitHub page and click the ★ Star and then ⑂ Fork buttons.
[ ] Clone a copy to your local machine with $ git clone git@github.com:YOUR-GITHUB-USER-NAME/n3m0.git
[ ] Make changes in the Readme.md file
[ ] Once you've made sure all your changes work correctly and committed all your changes, push your local changes back to github with $ git push -u origin master
[ ] Visit your fork on GitHub.com (https://github.com/YOUR-USER-NAME/n3m0) and create a pull request for your changes.
[ ] Make sure your pull request describes exactly what you changed and references this issue (include the issue number in the title like this: #5)
[ ] Please do not fix more than one issue at a time. Your pull request should only fix what is described in this issue.

Asking for help
We appreciate your effort in taking the time to work on this issue and help out the open source community and the foundation. If you need any help, feel free to ask below or in our gitter channel. We are always happy to help 😄

Hi, i'm a first timer. I would like to do this job (adding gitter badge), if it's possible ? Thanks

Hey @TheoCouture Go ahead 👍

Thanks for your contribution @TheoCouture Hope you had fun contributing and learned few things along the way! Feel free to join our gitter channel to know about more opportunities to contribute to opensource 🙂


SSL (htts) website detection and update of jqplot.

SSL (htts) website detection and update of jqplot.

Thanks!  Looks good to me.


Contents of Backup.bin

Could you please make available the backup.bin of A7RII or any other Sony camera?
That way we can help you analyze it and see what else can be done.

Use fwtool to unpack your camera's firmware update, the backups are in the 0110_backup folder (different files for different regions).
This reminds me that I should probably document the format somewhere, haven't done this yet. Look at /usr/kmod/backup.ko for the driver, it's not too hard to figure out what it does.

Would you be interested in collecting the backup.ko files from various models/regions, or does that not really matter?

Having a repository of backup.bin files, one for each camera model would be enough, I imagine.

It could be better to translate them into text files and compare side by side. No property names but only id's. But even this may be a good first step. Is not it?

I am a little confused here. Sorry for my ignorance.. Where can i find this backup.bin file ? and How to read/decode these .bin files?
Unpacked the latest A6000 firmware ( update 3.10) . I can't find a file named backup.bin in 0110_backup folder. Instead i have a SYSMUSASHI-DSLR folder and MainDest.txt . There is no backup.bin under that sub folder as well. However there are some other .bin files with name starting with CX7750 for eg CX77500_ALLLANG.bin

These are all the Backup.bin files for different regions (minus the settings that are already stored on the camera, like the serial number). I haven't documented the file format yet. Have a look at backup.ko if you want to reverse engineer it yourself.

no backup.ko on Nex6 firmware... Must be another driver!

Ok. Where can i find backup.ko ? /usr/kmod/backup.ko ? I don't have such a folder in firmware.tar_unpacked  or updater.img_unpacked folders.

Could you please attach the backup.ko here so that we can analyze it without having to extract it from a camera?

Yap, for 2nd gen devices, it's unidrv2.ko in initrd.img.
And no, we won't distribute any code by Sony here.

So...no backup.bin for A7M2 Japan-only ? Maybe it already changed into backup.ko ?

The Japanese-only backup file is included in every firmware. The filename ends with _J1.bin

// Have a look at backup.ko
No any names and/or hints in it regarding the meaning of the settings. It's only message receiver. At least i could not find anything it it. Main question for me is how you've able to find 0x3E0166 message type. But i understand that such information probably will not be published. In other hand it's probably be a bit easy to do some boring job in data collection and summarizing by somebody else but you.
Anyway your job is excellent! Thank you very much.

I hope this could help to somebody with other command research http://pastebin.com/cQH9x4z9

@Boldbrew thanks, that is helpful. I assume you extracted it from adjstctl.elf in nflasha15? At least, that's where I found it in rx100m4 firwmare v1.30. Based on the function that reads that table and dumps it with fprintf, I've added headings: http://pastebin.com/NY6uP15C

@lainy Most probably. But first fields used as bytes is not it? I mean 0x1, 0x1, 0x10, 0x1. And not all the meaning is clear for me. Anyway they pretty useless without clear payload data forming understanding.

@Boldbrew They are used as 16bit values, like:
fprintf(stdout, "adjust block: 0x%04x, code: 0x%04x OSAL ID: 0x%08lx cmdname: %s\n", block, code, osal_id, cmdname

Here is the relevant snippet:
loc_8A5A                          ; r4 is pointer to *next* struct
  LDR.W           R1, [R4,#-4]    ; so -4 is last 4bytes of current struct (set r1 = osal id)
  LDRH.W          R2, [R4,#-0x48] ; sizeof(struct) is 0x48, so r2 = block (first 16bits of struct)
  LDRH.W          R3, [R4,#-0x46] ; r3 = next halfword (code)
  STR             R1, [SP,#0xC+osal_id] ; store osal_id on stack
  MOVS            R1, #0x48
  MLA.W           R1, R1, index, R6 ; r1 = 0x48 * index, r6 not used
  ADDS            index, #1
  ADDS            R1, #4  ; offset of cmdname
  STR             R1, [SP,#0xC+cmdname_ptr] ; store cmdname ptr on stack
  LDR             R1, =stdout
  LDR             R0, [R1] ; stream
  LDR             R1, =aAdjustBlock0x0 ; "adjust block: 0x%04x, code: 0x%04x OSAL ID: 0x%08lx cmdname: %s\n"
  BLX             fprintf ; r0 is stdout
                          ; r1 is format string
                          ; r2 is block (16bit)
                          ; r3 is code (16bit)
                          ; stack contains osal_id (32bit) and cmdname (char*)

But you're right, not much use without a better understanding of it all. I've done some poking of my own on the Sony firmware to learn about the lens protocol, but always got distracted with other things...
Anyway, I suspect the data we want is in av-cam.bin (nflasha3), as it seems to handle the main camera functionality, but I could be way off :sweat_smile:

Well, the movie record time stuff seems to come from libmpr.so I think. At a glance, I see the hour, minute, and second values being referenced in a function which calls Backup_read() with them.

Looks so. A number of BKID_* constants could be found around. But they're not much promising from my point of view.

Hi guys, thank you so much for contributing to this project! I can't wait to pitch in my help and expand the capabilities! Can you share the method of decoding the binaries? I'm using Hexinator right now looking at adstctl.elf and I recognize some phrases. But how can I convert them into the snippet that you shared in pastebin? Thanks!

@MarcBlast I'm using IDA Pro to analyze the files, but a cheaper alternative is Hopper. There is also r2, but I've never used it.

Hi guys, sorry my less experience. I have a problem to search the Backup.bin file from my a6000 update. I have a 3.10 update, and i do uncompress the update with "fwtool" but just come back two folders (nexhack and resource) and another update installer. Where is the file Backup.bin? Whats wrong with my worksop for uncompress the update file?
Sorry for my english, im from Spain. Thanks for help!

@gtoni48 Look at the 2nd comment in this thread.

Hola @gtoni48 ,

Are you using this fwtool (http://github.com/ma1co/fwtool.py)?

Once unpacked, you will have 2 folder for A6000:
updater.img_unpacked
firmware.tar_unpacked   -- Firmware files are here
"Backup.bin" files : firmware.tar_unpacked\0110_backup\SYSMUSASHI-DSLR\DM_UUD3

Thanks, now i have fwtool.py from your link. But, how can i execute the software? Im really sorry and thanks again!

You must install python and then run from command line
python fwtool.py unpack -f FileUpdateNamexxx.exe

Before that command you'll need to run this one "pip install pycrypto"
On Tue, 10 May 2016 at 15:07 lh8511<EMAIL_ADDRESS>wrote:

You must install python and then run from command line
python fwtool.py unpack -f FileUpdateNamexxx.exe
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/ma1co/OpenMemories-Tweak/issues/1#issuecomment-218275395

I've added an option to fwtool.py to print all properties contained in a Backup.bin. There are now binaries for Windows and OS X, too.

@ma1co Thanks for print_backup command :-)

@ma1co thank you for your efforts!
So I opened my Backup.bin file with GIMP. Does anybody know who is that guy?

I did the same with another camera of the same model (AX100E) and found no face. Could it be some kind of PlayMemories avatar or something like that? In that case, the picture is of the previous owner that somehow survived factory resets?

@AlexDaniel Interesting... Could you upload the Backup.bin file to my dropbox so I can try to find out what this is?

@ma1co done.

@AlexDaniel The face is stored in backup property 0x00060009 which is used in av-cam.bin from file fc_data_frc.cpp. It seems to be related to face recognition if I'm not mistaken. Maybe it is the last face that was detected or tracked? If you want to play with this, can you try to make it change by trying all kinds of face detection modes? Also try tapping one of the detected faces to go to "face priority mode".


#159068263 Add GET entries/:id endpoint

What does this PR do?
Add an endpoint to retrieve a single journal entry by id
Description of Task to be completed

Add tests that fail for endpoint
Build minimum solution for test to pass
Iterate on tests

How should this be manually tested?
API endpoint has been deployed to: https://my-diary-api.herokuapp.com/api/v1/entries
Story with id=1 is available on server
What are the relevant pivotal tracker stories?
#159068263

Pull Request Test Coverage Report for Build 14

6 of 6 (100.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 12:
0.0%

Covered Lines:
22

Relevant Lines:
22

💛  - Coveralls


Looks good, can you target develop with this?

Hmm... thats not quite what I want. I'm going to close this and redo it so those other commits don't get mixed up in there


[SW-1064] Fix/transaction list composables

The name of the commit is incorrect.
feat: multichain account create modal

The extensions behaviour is weird. When i first open it it flashes and goes through the menus.
https://github.com/aeternity/superhero-wallet/assets/42606212/ace98729-44f9-418b-b310-8f6040e87919

Name registrations is not working. Its only loading up for ever.

Lgtm in general

@CharisSiarampalis please test the latest transactions widget when the view is in Multisig vaults and try to propose a TX with a multisig account and see if it appears in the wallet. There is a certain test that breaks the latest transaction widget @Liubov-crypto knows it well.

@CharisSiarampalis please test the latest transactions widget when the view is in Multisig vaults and try to propose a TX with a multisig account and see if it appears in the wallet. There is a certain test that breaks the latest transaction widget @Liubov-crypto knows it well.

I can see the multisig transaction on my wallet. I am not sure about the test that breaks it.

@konradodo please check it: Transactions from regular accounts are visible in multisig accounts:
https://github.com/aeternity/superhero-wallet/assets/69896204/fe226acf-474e-444d-8f48-10b3c969b506
Accounts discovery is broken in FF extension:

The issue with Transaction has been fixed.
I've registered a separate bug with Accounts discovery.
I found another issue: if I quick enter the amount I will be able to propose a Tx from vault with 0 balance. There is some delay in blocking Next button. This option is blocked on develop. @konradodo please check it:
https://github.com/aeternity/superhero-wallet/assets/69896204/7ca8262a-d445-4042-86f4-3c2ef079735e

Also I tried to claim a name, Tx was successful, but on this branch Tx is in Pending status for a long time . On dev branch it's completed. Maybe it's also worth to check:
https://github.com/aeternity/superhero-wallet/assets/69896204/da4e5f57-8692-48c5-b712-3ba5dce27d82

@konradodo  I found that issue with blocking Next button is reproducible on another PRs too, so I registered a separate issue for it: if I quick enter the amount I will be able to propose a Tx from vault with 0 balance. There is some delay in blocking Next button. This option is blocked on develop.


refactor(@vtmn/icons, @vtmn/assets): svgo config to add removeDimensions plugin

Changes description

I propose to add the removeDimensions plugin from svgo in order to have a better visualisation of our different icons & assets.
Before

After

Checklist

[x] Make sure you are requesting to pull a topic/feature/bugfix branch. Please, don't request directly from your main!
[x] Check commits & PR names matches our requested structure. It must follow the https://www.conventionalcommits.org pattern.
[x] Check your code additions will fail neither code linting checks.
[x] I have reviewed the submitted code.
[x] I have tested on related showcases.
[x] If it includes design changes, please ask for a review with a core team designer.

Does this introduce a breaking change?

No

I've checked by launching our different showcases and tested the icon font with some icons like "Button" with left/right & only icon. But, please if you can ensure this, that could be great. Thanks 🙏

@thibault-mahe as we didn't really use these heights and widths because we permit to use them in 16x16, 32x32 or 64x64, I think it is more related to viewBox and let our consumers choose the height/width with these specific attributes instead of choosing one of these by default.
For performance issues, I don't really know what this change will cause.


Create Databases

[ ] Create User Database storing logins
[ ] Create Match Database storing statistics and information relevent to that user and football
[ ] Connect Databases?


Incorrect Output

I ran the following on the DIV2K dataset.
python main.py --model EDSR --scale 2 --save edsr_x2 --n_resblocks 32 --n_feats 256 --res_scale 0.1 --reset --ext sep_reset --batch_size 10 --epoch 300 --patch_size 96  --n_GPUs 2 --chop
To test the model I ran
python main.py --data_test Demo --scale 2 --pre_train 'the directory' --test_only --save_results --n_resblocks 32 --n_feats 256
Now the output is not at all good as you can see from the images
Input :

Output :

Where am I going wrong ? I have not changed anything from the code and wanted to test the code as is on the DIV2K.
Thanks!

Hello,
It seems that you ignored the --res_scale 0.1 argument during your test.
Thank you.


input_datetime is an hour out on UK time

Make sure you are running the latest version of Home Assistant before reporting an issue.
You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:
Home Assistant release (hass --version):*
0.58.1
Python release (python3 --version):
3.6.0
Component/platform:
input_datetime
Description of problem:
In the UK we are currently on GMT.  We were on BST until the end of last month (one hour ahead).
Attempting to set the datetime to 19:00 sets it to 20:00.
Component seems to be correcting for an hour forward, as it is taking the time and unnecessarily converting it to BST.
Expected:
Component just accepts that I know what time want to set it to and doesn't convert it to BST, especially as I'm not in BST, and if I was I would still mean 19:00 if that's what I set.
Problem-relevant configuration.yaml entries and steps to reproduce:
input_datetime:
  masterbed_alarm_clock_time:
    name: Time
    has_date: false
    has_time: true
    initial: '06:15'

Include above in configuration.yaml whilst in UK timezone
Try to set the time to 19:00
Scratch head when it changes it to 20:00

Additional info:
(the above sarcasm is sense of humour, not annoyance 😃  )

@tinloaf - as discussed on the forums.

What exactly is the setting of your
homeassistant:
  time_zone: …

?

time_zone: Europe/London

:+1:

Same issue, also time_zone: Europe/London, running version 0.58.0
I'm running inside a docker-container  with a /etc/localtime:/etc/localtime:ro volume. gnu date from inside the container outputs, for example Sat Nov 25 19:53:25 GMT 2017 (which was correct).

Same issue here with time_zone: Pacific/Auckland

The front end time is one hour behind. The state is the correct time.

Had the chance to do some more investigation, looks like it's a chrome bug:-
In chrome dev console:
x = new Date(1970, 0, 1, 6, 0)
Thu Jan 01 1970 06:00:00 GMT+0000 (GMT)
x.toLocaleTimeString("en", {hour: "numeric", minute:"2-digit"})
"7:00 AM"
x = new Date(2000, 0, 1, 6, 0)
Sat Jan 01 2000 06:00:00 GMT+0000 (GMT)
x.toLocaleTimeString("en", {hour: "numeric", minute:"2-digit"})
"6:00 AM"

Also confirmed that the bug does not occur in firefox
Could be fixed by tweaking the default year/month/day in home-assistant-polymer/js/common/util/compute_state_display.js, but maybe that's too big a hammer...

HAH! it also works fine before 1968. According to https://en.wikipedia.org/wiki/British_Summer_Time we were GMT+1 all year between 1968 and 1971. I hate coding around timezones :(
Created chrome issue here: https://bugs.chromium.org/p/chromium/issues/detail?id=797548

(I have no doubt that you know what you are doing, this question is curiosity and not doubting your findings in any way)
Why does Chrome touch the time? Surely the whole point in a Web page is that I say "print 1900 on the screen in green coloured times new roman font" and that's what it does?

So, what home-assistant tells chrome to do is "print 19:00 in whatever the local style for displaying times is". 19:00 is an solid example, because that's displayed as 7:00 PM, but PM is an english abbreviation and has different equivalents elsewhere. See https://en.wikipedia.org/wiki/Date_and_time_representation_by_country if you want more info.

OK, so, (sorry to sound daft, as you can see I have no idea how this works) but...
Could we not take that option out of the browser's hands and just tell it to print 19:00? (like, send it a string instead)
My thinking being (bare with me):
I set up an input_datetime in my configuration.
I set the time to 1900
My frontend displays 2000 for the reason you've outlined.
BUT, in my devices list the state of the input_datetime is 1900, presumably because it's passed to the browser as a string rather than a 'time'?
Given that the frontend display is purely aesthetic, and given that I don't expect anybody will ever want it to show any time other than the one they've set could we not pass it as a string in the front end too?

That's potentially detrimental to some countries, depending on how they conventionally display times. To know whether it's safe, you've gotta start digging from that link I posted.
There's a simpler workaround HA can implement, which is to use the current year/month/day for the date they're required to include to get the browser to render a time, instead of 1970-01-01.  That particular selection has highlighted two, so it's not unreasonable to assume that any arbitrary selection will do the same. Today is less likely to be buggy :)
I'll see if I can't whip up a quick pr for that now actually, sure that's what you're actually after :)

Fair enough, just thinking out loud whilst simultaneously trying to learn something. Everyday is a school day, and as I type this the clock has just changed to midnight going in to Xmas day, so, ummm, Merry Christmas 😀😀
PS - a PR to fix would be an ace stocking filler 😂😂

Merry Xmas :)


Extension for MarginalLogDensities.jl

This PR adds an extension for MarginalLogDensities.jl, allowing one to call marginalize on a Model, resulting in a MarginalLogDensity from MarginalLogDensities.jl.
Example:
julia> using Turing, MarginalLogDensities

julia> @model function demo()
           x ~ Normal(0, 1)
           y ~ Normal(x, 1)
       end
demo (generic function with 2 methods)

julia> model = demo();

julia> # Marginalize out `x`.
       marginalized = marginalize(model, [@varname(x)]);

julia> # Compute the marginal log-density of `y = 0.0`.
       abs(marginalized([0.0]) - logpdf(Normal(0, √2), 0.0)) < 1e-1
true

Ref: #2398

A bit uncertain if we should export this marginalize or hide it for now.

Note that this will fail until https://github.com/ElOceanografo/MarginalLogDensities.jl/pull/36 has made its way to the public (i.e. v0.3.6). Not entirely certain why this hasn't happened yet (it is in the registry :confused: )

We need https://github.com/TuringLang/DynamicPPL.jl/pull/738 to go through before this will work :+1:


Process crash due to shared file access exceptions

I'm running a simple scenario where I add a bunch of items to a queue and having a trivial C# function process the messages. I'm only ever running on a single VM. Periodically, it seems the host process crashes, and I see the following in my eventlog.xml:

An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1961040831/ROOT
Process ID: 3472
Exception: System.IO.IOException
Message: The process cannot access the file 'D:\home\LogFiles\Application\Functions\Host\bd51ad17-edf1-4601-9cd8-dba2800743e4-6dd04c4de4.log' because it is being used by another process.
StackTrace:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
at System.IO.File.InternalAppendAllText(String path, String contents, Encoding encoding)
at System.IO.File.AppendAllText(String path, String contents)
at Microsoft.Azure.WebJobs.Script.FileTraceWriter.AppendLine(String line)
at Microsoft.Azure.WebJobs.Script.FileTraceWriter.Trace(TraceEvent traceEvent)
at Microsoft.Azure.WebJobs.Host.CompositeTraceWriter.InvokeTraceWriters(TraceEvent traceEvent)
at Microsoft.Azure.WebJobs.Host.CompositeTraceWriter.Trace(TraceEvent traceEvent)
at Microsoft.Azure.WebJobs.Host.Loggers.TraceWriterFunctionInstanceLogger.LogFunctionCompletedAsync(FunctionCompletedMessage message, CancellationToken cancellationToken)
at Microsoft.Azure.WebJobs.Host.Loggers.CompositeFunctionInstanceLogger.<LogFunctionCompletedAsync>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1.<TryExecuteAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueTriggerExecutor.<ExecuteAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Azure.WebJobs.Host.Queues.Listeners.QueueListener.<ProcessMessageAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Azure.WebJobs.Host.Timers.BackgroundExceptionDispatcher.<>c__DisplayClass1.<Throw>b__0()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

Seems like a recent regression since I didn't have this problem last week.

Thanks for reporting. Strange - we're doing locking around all our file writes: https://github.com/Azure/azure-webjobs-sdk-script/blob/master/src/WebJobs.Script/FileTraceWriter.cs#L89

It's also strange because it seems the callstack doesn't match the code, based on what you just showed me.  Specifically, the failure happens under AppendLine(), but in the code you just pointed me to AppendLine doesn't actually do any I/O.
Is there any way to find out which version has been deployed to Azure?

AppendLine was changed two releases ago (in this commit) to do buffering of writes to a ConcurrentQueue - it doesn't do file operations any more (we flush on background thread).
Perhaps you are running old bits in your site, perhaps via a private SiteExtension?

No, I'm definitely not using a SiteExtension, but it does appear that I'm somehow on an old build.  Is there an assembly version or something I can look at to confirm?  This is a private Antares deployment, so I don't know whether it affects public Azure, but my understanding is that all environments get the same bits.

Looks like I actually was using a private site extension.  There was a D:\home\SiteExtensions\Functions directory in my app and those were the bits being loaded. I stopped the site, deleted that directory, started the site again, and now I'm using the latest bits.  Not exactly sure how that happened, but for now I'm going to assume this is not a real product issue.

:)


Critical bugs in KOfam annotation step

I’m Iva (@ivagljiva), writing on behalf of myself and my colleague Kat (@Kekananen). We were looking into this pipeline for unrelated reasons when we stumbled across a couple of bugs. Unfortunately, after investigating and testing further to make sure the problem was not on our end, we determined that these bugs critically affect the KOfam annotations that MicrobeAnnotator reports. However, we have a fix ready to go, which we'll be posting in a PR linked to this issue.
To summarize, there are two major bugs in the MicrobeAnnotator code, specifically the part that filters the HMMER results from the initial search against the KOfam database (in the hmmsearch.py module). These bugs are:

only the first KO model in a given subset is used to obtain the bit score threshold for filtering out weak HMM hits within the hmmer_filter() function (by 'subset', we mean one of the MicrobeAnnotator_DB/kofam_data/profiles/*.model files). This means that a majority of the KOfam hits are filtered using the wrong threshold, leading to a large number of false positive annotations.
when identifying the best match to a given gene in the best_match_selector() function, a string comparison (rather than a numeric comparison) of the bit scores is done, leading to incorrect assignment of the best match in some cases.

We identified these issues by looking at the kofam_results/*kofam and kofam_results/*filt output files reported by the pipeline on our test genomes. We'll show the output from a publicly-available genome, Bradyrhizobium manausense BR3351 (NCBI RefSeq GCF_001440035.1), to help explain what is going on. Note that this output is based on the KOfam database downloaded with microbeannotator_db_builder from December 15, 2023.
Bug 1: filtering hits with incorrect threshold
In going through the *.filt output file, we noticed that some hits to a given KO were below the bit score threshold for that KO. For instance, here are a couple of hits to K00119, which has a (full) threshold of 510.70 in our KOfam database:
$ head microbeannotator_output/Bradyrhizobium_manausense_BR3351/kofam_results/Bradyrhizobium_manausense_BR3351_proteins.fasta.kofam.filt
#					----	full_sequence	----	----	best_1_domain	----	---	---	---	domain	number	estimation	---	---	description_of_target
# query_name	accession	target_name	accession	score_type	E-value	score	bias	E-value	score	bias	exp	reg	clu	ov	env	dom	rep	inc	description_of_target
# -----------	---------	----------	---------	----------	-------	-----	----	-------	-----	----	---	---	---	--	---	---	---	---	---------------------
6333	-	K00003	-	domain	1.4e-152	509.1	17.8	1.5e-152	508.9	17.8	1.0	1	0	0	1	1	1	1	homoserine dehydrogenase [EC:<IP_ADDRESS>]
1672	-	K00119	-	full	5.2e-23	80.6	8.9	4.2e-21	74.4	8.9	2.5	1	1	0	1	1	1	1	alcohol dehydrogenase (nicotinoprotein) [EC:<IP_ADDRESS>]
4973	-	K00119	-	full	2.3e-28	98.3	6.4	8.9e-28	96.3	6.4	1.8	1	1	0	1	1	1	1	alcohol dehydrogenase (nicotinoprotein) [EC:<IP_ADDRESS>]

The two hits have full bit scores of 80.6 and 98.3, respectively, both of which are much less than 510.70.
We looked into the code and realized that the following line from the hmmer_filter() function is the issue:
model_name = hmmsearch_result[0].query.name

The model name is extracted only from the initial HMM hit in the hmmsearch_result variable that is passed to the function. However, this variable contains the hits to multiple KOs, specifically all KOs in a given MicrobeAnnotator_DB/kofam_data/profiles/*.model, since each CPU thread works on an individual *.model file at a time (based on this part of the codebase). And since that model name is subsequently used to identify the bit score threshold used to filter all hits within hmmsearch_result, that means a majority of hits are filtered using the wrong threshold.
We added a print statement to the hmmer_filter() function that prints the model information each time a different model and threshold are loaded, like this:
def hmmer_filter(
    hmmsearch_result: List[hmmer.tbl.TBLRow]) -> List[hmmer.tbl.TBLRow]:
    # Initialize list with filtered results
    hmmsearch_result_filt = []
    # Get model name, threshold and score_type
    model_name = hmmsearch_result[0].query.name
    threshold = hmm_model_info[model_name].threshold
    score_type = hmm_model_info[model_name].score_type
    print(f"Filtering hits for model {model_name} with {score_type} threshold {threshold}")  ## ADDED
    (....)

When we ran the modified code on our test genome and saved the output to a log file, we counted only 27 of these print statements in the log:
$ grep "Filtering hits for model" microbeannotator.log
Filtering hits for model K03088 with domain threshold 97.23
Filtering hits for model K17081 with full threshold 414.0
Filtering hits for model K16413 with - threshold -
Filtering hits for model K00001 with domain threshold 379.83
Filtering hits for model K01363 with domain threshold 228.63
Filtering hits for model K21811 with full threshold 449.23
Filtering hits for model K21536 with full threshold 148.83
Filtering hits for model K25071 with full threshold 765.63
Filtering hits for model K00790 with domain threshold 172.07
Filtering hits for model K07037 with full threshold 218.93
Filtering hits for model K08071 with full threshold 378.6
Filtering hits for model K03446 with domain threshold 350.63
Filtering hits for model K09726 with full threshold 179.77
Filtering hits for model K24847 with full threshold 674.23
Filtering hits for model K05540 with full threshold 282.07
Filtering hits for model K17241 with full threshold 288.03
Filtering hits for model K19168 with domain threshold 24.5
Filtering hits for model K11354 with full threshold 460.53
Filtering hits for model K18130 with full threshold 215.6
Filtering hits for model K10022 with domain threshold 343.67
Filtering hits for model K16649 with full threshold 152.1
Filtering hits for model K08979 with full threshold 166.07
Filtering hits for model K18304 with full threshold 218.1
Filtering hits for model K19429 with full threshold 223.6
Filtering hits for model K23661 with full threshold 205.5
Filtering hits for model K15950 with full threshold 1043.53
Filtering hits for model K11061 with domain threshold 563.5
$ grep -c "Filtering hits for model" microbeannotator.log
27

The KOs in these print statements correspond to the first KO in each *.model file:
$ for f in MicrobeAnnotator_DB/kofam_data/profiles/common*.model MicrobeAnnotator_DB/kofam_data/profiles/prokaryote*.model MicrobeAnnotator_DB/kofam_data/profiles/independent*.model; do head $f | grep NAME; done
NAME  K00001
NAME  K01363
NAME  K03088
NAME  K07994
NAME  K17081
NAME  K00790
NAME  K18130
NAME  K18304
NAME  K09725
NAME  K08979
NAME  K15950
NAME  K11061
NAME  K16649
NAME  K19429
NAME  K23661
NAME  K02595
NAME  K11354
NAME  K24847
NAME  K05540
NAME  K03446
NAME  K17241
NAME  K09726
NAME  K11008
NAME  K13477
NAME  K25279
NAME  K15947
NAME  K15995

Going back to our initial example of K00119, that KO is stored within prokaryote_2.model, the first KO in that set is K02595, and K02595 has a threshold of 50.27, which explains why those two hits were included in our output.
$ grep K00119 MicrobeAnnotator_DB/kofam_data/profiles/*.model
MicrobeAnnotator_DB/kofam_data/profiles/prokaryote_2.model:NAME  K00119
$ head MicrobeAnnotator_DB/kofam_data/profiles/prokaryote_2.model | grep NAME
NAME  K02595
$ grep K02595 MicrobeAnnotator_DB/kofam_data/ko_list
K02595	50.27	full	all	1.000000	424	370	296	128	3.81	0.590	nitrogenase-stabilizing/protective protein

It seems like this happened because hmmer_filter() was initially designed to handle one KO at a time, maybe before the pipeline was multi-threaded using these *.model files?
This bug can cause both false positives (keeping hits that are too weak given the true threshold as defined by KEGG) and false negatives (removing strong hits given the true threshold) in the annotation results. We counted the number of false positive annotations produced for our test genome with a simple Python script that simply checks each hit in *.filt output file against the true bitscore threshold, and we found 2,852 false positive annotations out of 4,366 total annotations, which is 65%. Note that our tests on other, smaller genomes yielded false positive percentages of 47% to 62%.
Here is the script for checking the number of false positives:
#!/usr/bin/env python
# A script to count the number of false positive annotations from MicrobeAnnotator output
# Usage: python count_false_positives.py PATH_TO_ANNOTATION_OUTPUT_DIRECTORY
# see below for expected directory structure of PATH_TO_ANNOTATION_OUTPUT_DIRECTORY

import os
import sys

# command-line argument
PATH_TO_ANNOTATION_OUTPUT=sys.argv[1]     # path to the microbeannotator output files (each test genome has its own directory inside this one)

## CHANGE THESE INPUTS FOR ALTERNATIVE DIRECTORY SETUPS
PATH_TO_MICROBEANNOTATOR_DB="../../MicrobeAnnotator_DB/" # path to the database set up by microbeannotator_db_builder
# this list holds the name of each genome annotated by microbeannotater, to be used in building the file names later
# as in "Bradyrhizobium_manausense_BR3351/kofam_results/Bradyrhizobium_manausense_BR3351_proteins.fasta.kofam.filt"
genomes_to_process = ["Bradyrhizobium_manausense_BR3351"]

ko_list = os.path.join(PATH_TO_MICROBEANNOTATOR_DB, "kofam_data/ko_list")
ko_dict = {}
ko_file = open(ko_list, 'r')
for line in ko_file.readlines():
   line = line.strip()
   entries = line.split()
   if entries[0] == 'knum':
     continue
   else:
     if entries[1] == '-':
       ko_dict[entries[0]] = {'threshold': None, 'score_type': entries[2]}
     else:
       ko_dict[entries[0]] = {'threshold': float(entries[1]), 'score_type': entries[2]}

for g in genomes_to_process:
  filt_output_file = os.path.join(PATH_TO_ANNOTATION_OUTPUT, f"{g}/kofam_results/{g}_proteins.fasta.kofam.filt")
  incorrect_annotations = []
  with open(filt_output_file, 'r') as filt:
    for line in filt.readlines():
      if line.startswith("#"):
        continue;
      else:
        hit_data = line.strip().split()
        ko = hit_data[2]
        full_bitscore = float(hit_data[6])
        domain_bitscore = float(hit_data[9])
        if ko_dict[ko]['threshold'] is None:
          continue;
        if ko_dict[ko]['score_type'] == 'full' and full_bitscore < ko_dict[ko]['threshold']:
          incorrect_annotations.append((ko, full_bitscore, ko_dict[ko]['threshold']))
        elif ko_dict[ko]['score_type'] == 'domain' and domain_bitscore < ko_dict[ko]['threshold']:
          incorrect_annotations.append((ko, domain_bitscore, ko_dict[ko]['threshold']))
  print(f"for {g}, the number of filtered annotations with bitscore less than the threshold is {len(incorrect_annotations)}")

Note that we didn't quantify the false negatives since that is a more involved process.
Bug 2: string comparison of bit scores for 'best match' identification
We compared the *.kofam output to the *.filt output and realized that the annotation within *.filt was not always the hit with highest bit score selected from the possibilities within *.kofam for a given gene. For instance, one of the weak annotations for K00119 in *.filt was supposedly the 'best match' for gene 1672.

Clearly, the 'best match' should have been K00004, with the highest bit score of 407.9, and not K00119 with a bit score of 80.6.
We again added print statements and ran a portion of the best_match_selector() function in the Python terminal, with our test genome's *.kofam file as input, to see what was happening. Here is the code we ran:
# note that we set raw_results to be equal to a *.kofam output file first

with open(raw_results, 'r') as infile:
  best_matches = {};
  for line in infile:
    record = line.strip().split('\t');
    if record[0] not in best_matches:
      best_matches[record[0]] = [record[6], line];
    else:
      if record[6] > best_matches[record[0]][0]:
        print(f"old record has {best_matches[record[0]][0]}, new has {record[6]}")  ## ADDED
        print(f"type of old record: {type(best_matches[record[0]][0])}, type of new: {type(record[6])}")  ## ADDED
        best_matches[record[0]] = [record[6], line];

And here is one example from the printed output that showcases the problem:
old record has 117.8, new has 23.8
type of old record: <class 'str'>, type of new: <class 'str'>

This is a string comparison, not a numeric comparison, which means that the 'best match' is actually chosen based on alphabetical order of the bit scores rather than their relative value, and the two comparison types don't always yield the same result.
This bug causes weaker hits to sometimes be selected as the best match for a given gene, and it compounds the effect of the first bug. We wrote another script to count the number of times this happens in our test genomes. For B. manausense, we found 1,692 incorrect best matches out of 4,366 genes, for a percentage of ~39%. In our other tests, the percentage ranged from 21% to 35%.
Here is the script for checking the number of incorrect best matches:
#!/usr/bin/env python
# A script to count the number of false positive annotations from MicrobeAnnotator output
# Usage: python count_false_best_matches.py PATH_TO_ANNOTATION_OUTPUT_DIRECTORY
# see below for expected directory structure of PATH_TO_ANNOTATION_OUTPUT_DIRECTORY

import os
import sys

# command-line argument
PATH_TO_ANNOTATION_OUTPUT=sys.argv[1]     # path to the microbeannotator output files (each test genome has its own directory inside this one)

## CHANGE THESE INPUTS FOR ALTERNATIVE DIRECTORY SETUPS
PATH_TO_MICROBEANNOTATOR_DB="../../MicrobeAnnotator_DB/" # path to the database set up by microbeannotator_db_builder
# this list holds the name of each genome annotated by microbeannotater, to be used in building the file names later
# as in "Bradyrhizobium_manausense_BR3351/kofam_results/Bradyrhizobium_manausense_BR3351_proteins.fasta.kofam.filt"
genomes_to_process = ["Bradyrhizobium_manausense_BR3351",]

for g in genomes_to_process:
  kofam_output_file = os.path.join(PATH_TO_ANNOTATION_OUTPUT, f"{g}/kofam_results/{g}_proteins.fasta.kofam")
  filt_output_file = os.path.join(PATH_TO_ANNOTATION_OUTPUT, f"{g}/kofam_results/{g}_proteins.fasta.kofam.filt")
  # get the reported best matches
  filt_annotations = {}
  with open(filt_output_file, 'r') as filt:
    for line in filt.readlines():
      if line.startswith("#"):
        continue
      else:
        hit_data = line.strip().split()
        gene = hit_data[0]
        ko = hit_data[2]
        filt_annotations[gene] = ko
  # get the true best matches
  best_matches = {}
  with open(kofam_output_file, 'r') as kof:
    for line in kof.readlines():
      if line.startswith("#"):
        continue
      else:
        hit_data = line.strip().split()
        gene = hit_data[0]
        ko = hit_data[2]
        full_bitscore = float(hit_data[6])
        if gene not in best_matches:
          best_matches[gene] = (ko, full_bitscore)
        else:
          prev_bitscore = best_matches[gene][1]
          if full_bitscore > prev_bitscore:
            best_matches[gene] = (ko, full_bitscore)
  # compare true to reported
  genes_with_wrong_best_match = []
  for gene, best_match_tuple in best_matches.items():
    best_ko = best_match_tuple[0]
    if filt_annotations[gene] != best_ko:
      genes_with_wrong_best_match.append(gene)
  print(f"for {g}, the number of genes with the wrong best match annotation is {len(genes_with_wrong_best_match)}")

Conclusion
In summary, these bugs produce incorrect KOfam annotations (which may potentially be ameliorated by downstream annotations from Swissport, RefSeq, etc. in later steps of the pipeline, but can also still end up in the final annotation table). Luckily, these bugs ended up being straightforward to fix and accordingly, we are submitting a PR. We hope you will consider merging this PR to the main branch of MicrobeAnnotator.

@ivagljiva @Kekananen - I just came across your pull request while also digging through the code in an attempt to understand why the KOfam results from microbeannotator did not match results from other software. Major thanks for sharing these bug fixes, huge time saver!
I performed validation tests with the python scripts provided in your post, using the Parvimonas micra ATCC 33270 type strain genome (GCF_000154405.1) as an example.
count_false_positives.py results relevant to fix for Bug 1
Before fix (Using "cruizperez/MicrobeAnnotator" branch):
for GCF_000154405.1, the number of filtered annotations with bitscore less than the threshold is 318

After fix (Using "ivagljiva/MicrobeAnnotator/tree/bug_fixes" branch):
for GCF_000154405.1, the number of filtered annotations with bitscore less than the threshold is 0

False positive annotations
Total annotations
% false positives

Before fix
318
707
44.9%

After fix
0
943
0.0 %

count_false_best_matches.py results relevant to fix for Bug 2
Before fix (Using "cruizperez/MicrobeAnnotator" branch):
for GCF_000154405.1, the number of genes with the wrong best match annotation is 159

After fix (Using "ivagljiva/MicrobeAnnotator/tree/bug_fixes" branch):
for GCF_000154405.1, the number of genes with the wrong best match annotation is 0

Incorrect best matches
Total annotations
% incorrect best matches

Before fix
159
707
22.5%

After fix
0
943
0.0 %

Looks like everything in the pull request works as planned, relevant to commits 6602246 2d25a62 89f4c88

Wow. Thank you very much for your follow-up investigation and sharing your findings and confirmation, @bdaisley! You rock.

No problem at all! Happy to help ☺  Thanks for merging!


"Downloading and Processing Data" section in Collab Notebook doesn't work

I have an issue with collab notebook when i an trying to launch "Downloading and Processing Data".
When i am choosed the dozer scene and launched section i saw this error report:
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last):  File "/usr/local/bin/ns-download-data", line 5, in
from nerfstudio.scripts.downloads.download_data import entrypoint
File "/usr/local/lib/python3.10/dist-packages/nerfstudio/scripts/downloads/download_data.py", line 33, in
from nerfstudio.process_data import process_data_utils
File "/usr/local/lib/python3.10/dist-packages/nerfstudio/process_data/process_data_utils.py", line 25, in
import cv2
File "/usr/local/lib/python3.10/dist-packages/cv2/init.py", line 181, in
bootstrap()
File "/usr/local/lib/python3.10/dist-packages/cv2/init.py", line 153, in bootstrap
native_module = importlib.import_module("cv2")
File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
File "/usr/local/bin/ns-download-data", line 5, in
from nerfstudio.scripts.downloads.download_data import entrypoint
File "/usr/local/lib/python3.10/dist-packages/nerfstudio/scripts/downloads/download_data.py", line 33, in
from nerfstudio.process_data import process_data_utils
File "/usr/local/lib/python3.10/dist-packages/nerfstudio/process_data/process_data_utils.py", line 25, in
import cv2
File "/usr/local/lib/python3.10/dist-packages/cv2/init.py", line 181, in
bootstrap()
File "/usr/local/lib/python3.10/dist-packages/cv2/init.py", line 153, in bootstrap
native_module = importlib.import_module("cv2")
File "/usr/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: numpy.core.multiarray failed to import
Data Processing Succeeded!
The section "Start Training" also doesn't work and returns this output:
Error: Data processing did not complete
Please re-run Downloading and Processing Data, or view the FAQ for more info.

I hit this too. I was able to fix this by running !pip install "numpy<2.0" in a new cell


[Enhancement] Improve accessibility for TouchEffect

Summary
Android :
Set View.Clickable or View.LongClickable to true only if we set Command or LongPressCommand respectively.
Both are currently set to true.
iOS :
Set UIAccessibilityTrait.Button when we have Command or LongPressCommand.
API Changes

Intended Use Case

Who Will Do The Work?

[ ] I am willing to take this on myself
[X] Just putting this out there for someone to pick up

Thanks! However, we are no longer adding new features to Xamarin Community Toolkit, focusing on the .NET MAUI Community Toolkit.
Please open a New Feature Discussion to implement this feature in the .NET MAUI Community Toolkit.
I've posted more information about the Future Of Xamarin Community Toolkit here: https://devblogs.microsoft.com/xamarin/the-future-of-xamarin-community-toolkit/?WT.mc_id=mobile-0000-bramin


validation of parameters with the same name

We don't have an integration test for timestamps. Probably, it's worth adding.


Object reference not set to an instance of an object

Plugin name and version
com.microsoft.azure.azure-functions-maven-plugin:0.1.10-SNAPSHOT
Steps to reproduce the problem

Generate a scaffolding through the Azure Functions Maven Archetype
mvn archetype:generate -DarchetypeGroupId=com.microsoft.azure -DarchetypeArtifactId=azure-functions-archetype
mvn package azure-functions:package
mvn azure-functions:run
Ctrl-C to exist
Run mvn azure-functions:run again

Looks like this is an issue related with the function cli.

Hi @bsaby I cannot repro this bug within the newest azure-functions-core-tools-2.0.1-beta.28.
Could you please check if this issue still occurs on your side?

Version: azure-functions-core-tools:2.0.1-beta.28
On my side, repro in git bash, but it works on my PowerShell. It doesn't seem to be our plugin's bug,  close it.


Do "Read" requests allow filtering on non-key fields?

In implementing a P4Runtime server, I'm trying to understand the intended behavior of the Read request when a Entity includes a value for a non-key field. For example, suppose that an Entity in the request is a TableEntry whose metadata member is nonempty. Does this select only the table entries whose metadata is equal to the one in the Entity? I think this is the intention based on https://github.com/p4lang/p4runtime/issues/229#issuecomment-517868612, but I could not find good support for this in the text.
This would lead to some odd corner cases, for example it would be possible to filter for table entries with a particular nonzero idle_timeout_ns, but not those with a zero idle_timeout_ns.
(I read P4Runtime 1.3.0 for this purpose.)
Guidance or textual clarification is appreciated.

This section describes the filtering mechanism supported for wildcard reads on table entries. Not all fields are supported, e.g. idle_timeout_ns.
For the metadata field:

metadata: If default (empty byte string), all entries from the specified table will be considered. Otherwise, results will be filtered based on the provided metadata value.

Because of Protobuf 3 semantics, it is not possible to request all entries for which metadata is unset / empty. In a lot of cases, the zero-value for a field is reserved and is not a valid value. For example, a priority of 0 for a ternary entry is not valid, so there is no ambiguity for the priority field. When we have a field for which we clearly need to distinguish between unset and the zero-value, we usually wrap the individual field in its own Protobuf message (and example would be the index field for indirect counters & meters). However, in the case of metadata, we did not deem it necessary at the time.

OK, thanks! The explanation was there, I just did not read carefully enough.
I don't think anything is needed here, so I'll close this.


Add possibility to create queue_group

This PR add the possibility to create queue_group in sendmail. We allready use it on our puppet with success.

Thanks for your contribution!
I've looked into the change. Since queue groups allow only a limited number of arguments and some of them have a data type restriction (R= must be an integer, P= must be an absolute path), I would rather have specific arguments and data types instead of a generic args parameter (cf. sendmail::mc::daemon_options).
What do you think?


NavigationController not showing up

Hi,
When I'm using presentr to display my VC, I'm losing my navigationController.
It's really annoying, I can see the navigation controller on my storyboard, but as soon as I'm using presentr (I'm using it to display a VC as a side bar menu, but still want to keep hierarchy in my VC).
Is there a way to specify the navigation controller ?
Btw, I'm using "customPresentViewController" func to display my VC.

Hi @FabylouV !
If you want to present the hierarchy Navigation Controller -> Your View Controller you must present the navigation controller (the one from your storyboard) which includes your view controller as it's child.
Or wrap your view controller in code, inside a Navigation Controller, and then present that.

Could you give me an example of each, because I'm really not following :x
I tried to do
VC1.customPresentViewController(presenter, viewController: VC2.navigationController!, animated: true, completion: nil)
But the VC2.navigationController is nil

I fixed my issue.
I had a big misunderstanding between displaying modally and dsplaying using hierarchy.
So I displayed my SideBar modally, but I pass the viewController in parameters so once I clicked on an item from my sideBar, I could load the view using hierarchy from my previous navigationController.

Awesome, yeah sounds about right.
Glad you fixed it. 👍


mvn clean install -PbuildDocker don't work

It is my error info:

Step 9/10 : EXPOSE ${EXPOSED_PORT}
│[INFO] ------------------------------------------------------------------------
│[INFO] Reactor Summary:
│[INFO]
│[INFO] spring-petclinic-microservices ..................... SUCCESS [  0.246 s]
│[INFO] spring-petclinic-admin-server ...................... FAILURE [ 10.753 s]
│[INFO] spring-petclinic-monitoring ........................ SKIPPED
│[INFO] spring-petclinic-customers-service ................. SKIPPED
│[INFO] spring-petclinic-vets-service ...................... SKIPPED
│[INFO] spring-petclinic-visits-service .................... SKIPPED
│[INFO] spring-petclinic-config-server ..................... SKIPPED
│[INFO] spring-petclinic-discovery-server .................. SKIPPED
│[INFO] spring-petclinic-api-gateway ....................... SKIPPED
│[INFO] spring-petclinic-tracing-server .................... SKIPPED
│[INFO] ------------------------------------------------------------------------
│[INFO] BUILD FAILURE
│[INFO] ------------------------------------------------------------------------
│[INFO] Total time: 11.951 s
│[INFO] Finished at: 2018-07-11T10:30:27+08:00
│[INFO] Final Memory: 75M/651M
│[INFO] ------------------------------------------------------------------------
│[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:0.4.13:build (default) on project spri
│ng-petclinic-admin-server: Exception caught: EXPOSE requires at least one argument -> [Help 1]
│[ERROR]
│[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
│[ERROR] Re-run Maven using the -X switch to enable full debug logging.
│[ERROR]
│[ERROR] For more information about the errors and possible solutions, please read the following artic
│les:
│[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
│[ERROR]
│[ERROR] After correcting the problems, you can resume the build with the command
│[ERROR]   mvn  -rf :spring-petclinic-admin-server

It seems like the expose require an argument.What it should be?
This is the dockerfile:

I have tried to set EXPOSE_PORT to 22 and 8899 which doesn't work.
This error info:

[ERROR] Failed to execute goal com.spotify:docker-maven-plugin:0.4.13:build (default) on project spri
│ng-petclinic-admin-server: Exception caught: Request error: POST unix://localhost:80/build?buildargs=
│%7B%22ARTIFACT_NAME%22%3A%22spring-petclinic-admin-server-1.5.9%22%2C%22EXPOSED_PORT%22%3A%229090%22%
│7D&t=mszarlinski/spring-petclinic-admin-server: 500: HTTP 500 Internal Server Error -> [Help 1]
│[ERROR]

Thanks!

Did you run the Docker daemon?

Yes,I have

This is my question in stackoverflow.
https://stackoverflow.com/questions/51277418/mvn-clean-install-pbuilddocker-dont-work
Thanks!Wait for your answer!

I've create the #82 to upgrade the Docker maven plugin but the current one is working on my macbook and Docker 18.05.0-ce-rc.

There is already an issue about builder Docker image with Maven: #64
Do you have checked the answers?

No feedback. Please reopen this issue if your problem persists.


⚠ Bofrost.lu has degraded performance

In ac34ebb, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 2803 ms

Resolved: Bofrost.lu performance has improved in 82807ed after 25 minutes.