add inscription_indexer.md

Project Abstract

Please replace these instructions with a brief description of your project summarising key points (1-2 paragraphs).
If your application is a follow-up to a previous grant, please mention which one in the first line of the abstract and include a link to previous pull requests if applicable.

Grant level

[ ] Level 1:  Up to $10,000, 2 approvals
[x] Level 2:  Up to $30,000, 3 approvals
[ ] Level 3:  Unlimited, 5 approvals (for >$100k: Web3 Foundation Council approval)

Application Checklist

[x] The application template has been copied and aptly renamed (project_name.md).
[x] I have read the application guidelines.
[x] Payment details have been provided (Polkadot AssetHub (DOT, USDC & USDT) address in the application and bank details via email, if applicable).
[x] I am aware that, in order to receive a grant, I (and the entity I represent) have to successfully complete a KYC/KYB check.
[x] The software delivered for this grant will be released under an open-source license specified in the application.
[x] The initial PR contains only one commit (squash and force-push if needed).
[x] The grant will only be announced once the first milestone has been accepted (see the announcement guidelines).
[ ] I prefer the discussion of this application to take place in a private Element/Matrix channel. My username is: @_______:matrix.org (change the homeserver if you use a different one)

I have read and hereby sign the Contributor License Agreement.

I have read and hereby sign the Contributor License Agreement.

Hey @jyaymusk ,
thank you and your team for the application.
As you correctly mentioned in "Ecosystem Fit" there are other teams who had come up with this idea already a long time ago. I'm looking at RMRK here. As I see it the RMRK team has even deprecated this way of minting by now and is using other ways. On Polkadot / Kusama they now use the uniques pallet for NFT creation and interaction.
I think it would make sense for you to research if there really is a need for this when the history so far shows that it has been successfully delivered and published but still got discontinued. Just for the sake of creating this solution with a different tech stack most likely won't appeal to the grants committee that ultimately decides on applications.
I would suggest to build on top of existing open source solutions in any case. See here the different specs by the RMRK ream.
Let me know what you think.

Hey @jyaymusk , thank you and your team for the application.
As you correctly mentioned in "Ecosystem Fit" there are other teams who had come up with this idea already a long time ago. I'm looking at RMRK here. As I see it the RMRK team has even deprecated this way of minting by now and is using other ways. On Polkadot / Kusama they now use the uniques pallet for NFT creation and interaction.
I think it would make sense for you to research if there really is a need for this when the history so far shows that it has been successfully delivered and published but still got discontinued. Just for the sake of creating this solution with a different tech stack most likely won't appeal to the grants committee that ultimately decides on applications.
I would suggest to build on top of existing open source solutions in any case. See here the different specs by the RMRK ream.
To now talk about your plans for an indexer, I think there is already an indexer well capable of the needs you describe. I recommend to check out Subsquid and re-evaluate your wish of creating something completely from the ground up.
Let me know what you think.

Thank you @PieWol , Thank you very much for your thorough response. We understand that no product can fully meet all users' needs, and we also appreciate the work done by Subsquid, which is truly remarkable and suitable for larger-scale applications. We understand that the scope of data parsing in Inscriptions is not broad, but rather, the key lies in determining the scope, data structures, and rules for data entry into the database. In other words, the challenge lies not in data parsing, but in establishing a standardized approach.
We are focused on how to quickly create products on Polkadot similar to BRC20, or even BRC404. From these perspectives, DOTA has been successful in meeting the needs of these users. If you pay attention to the expectations of our community and the efforts of our team, I believe you will share the same thoughts. Inscriptions Index pays even more attention to the rules of ledgers and tokens, and how related data is entered into the database according to these rules, as well as how to ensure the consistency of community data – all of which are crucial for consensus. Everyone has data stored locally in MySQL, so how to quickly verify that the ledger is consistent with the entire network, and so on, are still many details that need to be addressed.
Currently, we have established the dot-20 specification for indexing in DOTA. This allows developers to create products based on this protocol, similar to ERC20-based products like OpenSea, without needing to rely on smart contracts. To achieve this, the indexer needs to parse and operate on data according to the requirements of dot-20. This requires a new indexer that is dedicated to this direction. We know that off-chain products are more likely to meet user experience expectations, which is a direction worth exploring.
We hope that the creation of this indexer will not only benefit DOTA but also be detached and become a new Inscriptions Indexer specification and a public infrastructure for Polkadot, where anyone can contribute interesting ideas or pull requests, eliminating the need for redundant development. A unified standard will also facilitate information exchange between Inscriptions platforms. Inscriptions is not a last resort; it is a new way of thinking that may drive entirely new interactions between on-chain and off-chain, as well as potentially bringing new technical solutions and interesting products. We should prepare for this. In fact, almost all public chains are doing similar things. A unified specification for the Inscriptions Indexer will also bring the benefit of many people participating in collective construction, ensuring security, and making integration with traditional CEX more efficient and easier to adopt.

Hey @jyaymusk , thank you and your team for the application.
As you correctly mentioned in "Ecosystem Fit" there are other teams who had come up with this idea already a long time ago. I'm looking at RMRK here. As I see it the RMRK team has even deprecated this way of minting by now and is using other ways. On Polkadot / Kusama they now use the uniques pallet for NFT creation and interaction.
I think it would make sense for you to research if there really is a need for this when the history so far shows that it has been successfully delivered and published but still got discontinued. Just for the sake of creating this solution with a different tech stack most likely won't appeal to the grants committee that ultimately decides on applications.
I would suggest to build on top of existing open source solutions in any case. See here the different specs by the RMRK ream.
To now talk about your plans for an indexer, I think there is already an indexer well capable of the needs you describe. I recommend to check out Subsquid and re-evaluate your wish of creating something completely from the ground up.
Let me know what you think.

Thank you @PieWol , Thank you very much for your thorough response. We understand that no product can fully meet all users' needs, and we also appreciate the work done by Subsquid, which is truly remarkable and suitable for larger-scale applications. We understand that the scope of data parsing in Inscriptions is not broad, but rather, the key lies in determining the scope, data structures, and rules for data entry into the database. In other words, the challenge lies not in data parsing, but in establishing a standardized approach.
We are focused on how to quickly create products on Polkadot similar to BRC20, or even BRC404. From these perspectives, DOTA has been successful in meeting the needs of these users. If you pay attention to the expectations of our community and the efforts of our team, I believe you will share the same thoughts. Inscriptions Index pays even more attention to the rules of ledgers and tokens, and how related data is entered into the database according to these rules, as well as how to ensure the consistency of community data – all of which are crucial for consensus. Everyone has data stored locally in MySQL, so how to quickly verify that the ledger is consistent with the entire network, and so on, are still many details that need to be addressed.
Currently, we have established the dot-20 specification for indexing in DOTA. This allows developers to create products based on this protocol, similar to ERC20-based products like OpenSea, without needing to rely on smart contracts. To achieve this, the indexer needs to parse and operate on data according to the requirements of dot-20. This requires a new indexer that is dedicated to this direction. We know that off-chain products are more likely to meet user experience expectations, which is a direction worth exploring.
We hope that the creation of this indexer will not only benefit DOTA but also be detached and become a new Inscriptions Indexer specification and a public infrastructure for Polkadot, where anyone can contribute interesting ideas or pull requests, eliminating the need for redundant development. A unified standard will also facilitate information exchange between Inscriptions platforms. Inscriptions is not a last resort; it is a new way of thinking that may drive entirely new interactions between on-chain and off-chain, as well as potentially bringing new technical solutions and interesting products. We should prepare for this. In fact, almost all public chains are doing similar things. A unified specification for the Inscriptions Indexer will also bring the benefit of many people participating in collective construction, ensuring security, and making integration with traditional CEX more efficient and easier to adopt.

Nearly 50 people are currently involved in the construction of this indexer. Due to the fact that the dota product is still in its early stages and requires absolute security, we only allow them to participate in testing and propose ideas.

Hey @jyaymusk , thank you and your team for the application.
As you correctly mentioned in "Ecosystem Fit" there are other teams who had come up with this idea already a long time ago. I'm looking at RMRK here. As I see it the RMRK team has even deprecated this way of minting by now and is using other ways. On Polkadot / Kusama they now use the uniques pallet for NFT creation and interaction.
I think it would make sense for you to research if there really is a need for this when the history so far shows that it has been successfully delivered and published but still got discontinued. Just for the sake of creating this solution with a different tech stack most likely won't appeal to the grants committee that ultimately decides on applications.
I would suggest to build on top of existing open source solutions in any case. See here the different specs by the RMRK ream.
To now talk about your plans for an indexer, I think there is already an indexer well capable of the needs you describe. I recommend to check out Subsquid and re-evaluate your wish of creating something completely from the ground up.
Let me know what you think.

Thank you @PieWol , Thank you very much for your thorough response. We understand that no product can fully meet all users' needs, and we also appreciate the work done by Subsquid, which is truly remarkable and suitable for larger-scale applications. We understand that the scope of data parsing in Inscriptions is not broad, but rather, the key lies in determining the scope, data structures, and rules for data entry into the database. In other words, the challenge lies not in data parsing, but in establishing a standardized approach.
We are focused on how to quickly create products on Polkadot similar to BRC20, or even BRC404. From these perspectives, DOTA has been successful in meeting the needs of these users. If you pay attention to the expectations of our community and the efforts of our team, I believe you will share the same thoughts. Inscriptions Index pays even more attention to the rules of ledgers and tokens, and how related data is entered into the database according to these rules, as well as how to ensure the consistency of community data – all of which are crucial for consensus. Everyone has data stored locally in MySQL, so how to quickly verify that the ledger is consistent with the entire network, and so on, are still many details that need to be addressed.
Currently, we have established the dot-20 specification for indexing in DOTA. This allows developers to create products based on this protocol, similar to ERC20-based products like OpenSea, without needing to rely on smart contracts. To achieve this, the indexer needs to parse and operate on data according to the requirements of dot-20. This requires a new indexer that is dedicated to this direction. We know that off-chain products are more likely to meet user experience expectations, which is a direction worth exploring.
We hope that the creation of this indexer will not only benefit DOTA but also be detached and become a new Inscriptions Indexer specification and a public infrastructure for Polkadot, where anyone can contribute interesting ideas or pull requests, eliminating the need for redundant development. A unified standard will also facilitate information exchange between Inscriptions platforms. Inscriptions is not a last resort; it is a new way of thinking that may drive entirely new interactions between on-chain and off-chain, as well as potentially bringing new technical solutions and interesting products. We should prepare for this. In fact, almost all public chains are doing similar things. A unified specification for the Inscriptions Indexer will also bring the benefit of many people participating in collective construction, ensuring security, and making integration with traditional CEX more efficient and easier to adopt.

Our plan is to create a parachain focused on inscriptions, going beyond just interacting with the chain through 'remark_with_event'. We hope that the Inscriptions platform can interact with the chain at a system level simultaneously. For instance, besides sending 'remark_with_event', we aim to enable the Inscriptions platform to send calldata or other types of memos to the chain. These calldata or memos can trigger operations on the chain, allowing for deeper interaction between the chain and the Inscriptions platform. The foundation for realizing this vision lies in having a certain type of Inscriptions indexer and rules. This is very appealing as it can meet the needs of different users. One can imagine that in the future, operations that would typically require a significant amount of gas fees could be executed with just a simple memo, saving a lot of costs for users. Operations that would typically occur on-chain, such as in social and gaming contexts, could be combined on-chain and off-chain, greatly optimizing the user experience, among other benefits.

Hey @jyaymusk , thank you and your team for the application.
As you correctly mentioned in "Ecosystem Fit" there are other teams who had come up with this idea already a long time ago. I'm looking at RMRK here. As I see it the RMRK team has even deprecated this way of minting by now and is using other ways. On Polkadot / Kusama they now use the uniques pallet for NFT creation and interaction.
I think it would make sense for you to research if there really is a need for this when the history so far shows that it has been successfully delivered and published but still got discontinued. Just for the sake of creating this solution with a different tech stack most likely won't appeal to the grants committee that ultimately decides on applications.
I would suggest to build on top of existing open source solutions in any case. See here the different specs by the RMRK ream.
To now talk about your plans for an indexer, I think there is already an indexer well capable of the needs you describe. I recommend to check out Subsquid and re-evaluate your wish of creating something completely from the ground up.
Let me know what you think.

Thank you @PieWol , Thank you very much for your thorough response. We understand that no product can fully meet all users' needs, and we also appreciate the work done by Subsquid, which is truly remarkable and suitable for larger-scale applications. We understand that the scope of data parsing in Inscriptions is not broad, but rather, the key lies in determining the scope, data structures, and rules for data entry into the database. In other words, the challenge lies not in data parsing, but in establishing a standardized approach.
We are focused on how to quickly create products on Polkadot similar to BRC20, or even BRC404. From these perspectives, DOTA has been successful in meeting the needs of these users. If you pay attention to the expectations of our community and the efforts of our team, I believe you will share the same thoughts. Inscriptions Index pays even more attention to the rules of ledgers and tokens, and how related data is entered into the database according to these rules, as well as how to ensure the consistency of community data – all of which are crucial for consensus. Everyone has data stored locally in MySQL, so how to quickly verify that the ledger is consistent with the entire network, and so on, are still many details that need to be addressed.
Currently, we have established the dot-20 specification for indexing in DOTA. This allows developers to create products based on this protocol, similar to ERC20-based products like OpenSea, without needing to rely on smart contracts. To achieve this, the indexer needs to parse and operate on data according to the requirements of dot-20. This requires a new indexer that is dedicated to this direction. We know that off-chain products are more likely to meet user experience expectations, which is a direction worth exploring.
We hope that the creation of this indexer will not only benefit DOTA but also be detached and become a new Inscriptions Indexer specification and a public infrastructure for Polkadot, where anyone can contribute interesting ideas or pull requests, eliminating the need for redundant development. A unified standard will also facilitate information exchange between Inscriptions platforms. Inscriptions is not a last resort; it is a new way of thinking that may drive entirely new interactions between on-chain and off-chain, as well as potentially bringing new technical solutions and interesting products. We should prepare for this. In fact, almost all public chains are doing similar things. A unified specification for the Inscriptions Indexer will also bring the benefit of many people participating in collective construction, ensuring security, and making integration with traditional CEX more efficient and easier to adopt.

Our plan is to create a parachain focused on inscriptions, going beyond just interacting with the chain through 'remark_with_event'. We hope that the Inscriptions platform can interact with the chain at a system level simultaneously. For instance, besides sending 'remark_with_event', we aim to enable the Inscriptions platform to send calldata or other types of memos to the chain. These calldata or memos can trigger operations on the chain, allowing for deeper interaction between the chain and the Inscriptions platform. The foundation for realizing this vision lies in having a certain type of Inscriptions indexer and rules. This is very appealing as it can meet the needs of different users. One can imagine that in the future, operations that would typically require a significant amount of gas fees could be executed with just a simple memo, saving a lot of costs for users. Operations that would typically occur on-chain, such as in social and gaming contexts, could be combined on-chain and off-chain, greatly optimizing the user experience, among other benefits.

We acknowledge the pioneering work done by the RMRK team in the early stages, which was truly ahead of its time. However, they clearly did not anticipate that inscriptions would become such a hot trend in the blockchain market and serve as a complement to blockchain technology, as evidenced by the current trend across all public chains. Now, there is much to be done, extending beyond just NFTs and branching out into various other scenarios. This includes adhering to the same protocols as other public chains' inscriptions, innovating new protocols, and following the direction of the latest inscription products.
We aim to establish a reliable and secure index on DOTA, allowing everyone to use it with confidence and enabling more applications to be built on top of this indexer. This indexer not only parses transactions and stores them in MySQL but, more importantly, filters data based on protocol rules, executes rules (protocols), and aggregates data. This also means that we are not simply a web scraper for native chain data.
While we hope that Subsquid's technology can assist us in this endeavor, it is not currently necessary. We have a team of seasoned engineers familiar with Substrate, with five years of experience in the Polkadot ecosystem. In simple terms, what DOTA is doing is not in conflict with what other projects in the ecosystem are doing; rather, it complements them and innovates based on the excellent ideas of these projects. The unprecedented transaction volume during the DOTA minting period on the Polkadot chain is evidence enough of the market demand and the need for deeper technical exploration within the Polkadot ecosystem in this field.
The technical exploration in the field of inscriptions should not stagnate in the past.

Hey @jyaymusk,
can you please refrain from quoting yours answers while still answering another post? It just creates this huge wall of text which is unpleasant to read.
I read this post on the polkadot forum. Have you tried to get funding from the treasury too? What was the outcome?
You say that subsquid is not suitable for your needs and you need to develop something yourself. Can you please concisely list what exactly subsquid is missing?
Thank you.

Hey @jyaymusk hope you are doing well. Do you have any updates to share?

Hey @jyaymusk hope you are doing well. Do you have any updates to share?

Thank you very much for your patient response and cautious attitude towards the Polkadot grant funding. Indeed, over the past month, we have initiated a proposal for Polkadot Treasury funds, and we are very grateful for the extensive participation in this discussion, which was beyond our expectations. This is also one of the few Treasury proposals in Polkadot's history to have such extensive participation and involvement of DOT holders.
We are disappointed that we did not secure the funding, but this will not deter us from continuing our determination to build the ecosystem and products. We have made significant progress in engaging and communicating with the community and are confident that we will soon launch another proposal. We are confident that we will gain understanding and support. Thank you!

Hey @jyaymusk hope you are doing well. Do you have any updates to share?

This project is an indexer for dota. It has its own protocol to comply with, such as dot20, a protocol like ERC20, which also determines the specificity of the function of the indexer itself. This new indexer is now in testing and entering its final stages. Its completion means a comprehensive upgrade of Dota and more community products will be built on it, which can directly help the community. We also know that subsquid can obtain on-chain data, but it is different from our direction. We focus more on the development of indexers based on specific protocols, and data acquisition and mining based on protocols. This is the code for our indexer to obtain and filter data on the chain. If you have time, you can take a look and provide suggestions. It may also be helpful to our further discussions.

recheck

see above


bin size 257 cannot run on GPU

I know there are a couple other issues that mention this problem, but it's gotten messy with suggestions it's related to categorical_feature setting and other stuff. Here is clean MRE.
d9a96c90cb479cef87047ba20517d97982b563eb
lgb257.pkl.zip
import pickle
model, X, y, kwargs = pickle.load(open(lgb257.pkl, "rb"))
model.fit(X, y, **kwargs)

FYI a model.get_params() shows:
params = {'boosting_type': 'gbdt', 'class_weight': None, 'colsample_bytree': 0.8, 'importance_type': 'gain',
          'learning_rate': 0.5, 'max_depth': 6, 'min_child_samples': 1, 'min_child_weight': 1.0, 'min_split_gain': 0.0,
          'n_estimators': 100, 'n_jobs': 8, 'num_leaves': 64, 'objective': 'binary', 'random_state': 1234,
          'reg_alpha': 0.0, 'reg_lambda': 1.0, 'silent': True, 'subsample': 0.7, 'subsample_for_bin': 200000,
          'subsample_freq': 1, 'pred_gap': None, 'pred_periods': None, 'max_bin': 255, 'scale_pos_weight': 1.0,
          'max_delta_step': 0.0, 'min_data_in_bin': 1, 'seed': 1234, 'early_stopping_limit': None, 'device_type': 'gpu',
          'gpu_device_id': 0, 'gpu_platform_id': 0, 'gpu_use_dp': True, 'feature_fraction_seed': 1235,
          'bagging_seed': 1236, 'num_threads': 8, 'num_class': 1, 'verbose': -1, 'categorical_feature': ''}

and FYI here is kwargs:

[LightGBM] [Warning] num_threads is set=8, n_jobs=8 will be ignored. Current value: num_threads=8
[LightGBM] [Warning] seed is set=1234, random_state=1234 will be ignored. Current value: seed=1234
/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/basic.py:1586: UserWarning: Using categorical_feature in Dataset.
  warnings.warn('Using categorical_feature in Dataset.')
/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/basic.py:1590: UserWarning: categorical_feature in Dataset is overridden.
New categorical_feature is []
  'New categorical_feature is {}'.format(sorted(list(categorical_feature))))
/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/basic.py:1108: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.
Please use categorical_feature argument of the Dataset constructor to pass this parameter.
  .format(key))
[LightGBM] [Fatal] bin size 257 cannot run on GPU
Traceback (most recent call last):
  File "/home/jon/h2oai.fullcondatest/h2oaicore/lgb257.py", line 18, in <module>
    model.fit(X, y, **kwargs)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/sklearn.py", line 867, in fit
    callbacks=callbacks, init_model=init_model)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/sklearn.py", line 637, in fit
    callbacks=callbacks, init_model=init_model)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/engine.py", line 230, in train
    booster = Booster(params=params, train_set=train_set)
  File "/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/basic.py", line 2104, in __init__
    ctypes.byref(self.handle)))
  File "/home/jon/minicondadai/lib/python3.6/site-packages/lightgbm_gpu/basic.py", line 52, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: bin size 257 cannot run on GPU

Running
model.fit(X, y)

fails same way, but I'm unsure for sklearn API if it is using 'auto' for categorical_feature then.

Here is more minimal MRE:
import pickle
X, y = pickle.load(open("/home/jon/lgb257b.pkl", "rb"))

params = dict(categorical_feature='', device_type='gpu', gpu_device_id=0, gpu_platform_id=0, gpu_use_dp=True, min_data_in_bin=1)
model = lgb.LGBMClassifier(**params)
model.fit(X, y, categorical_feature='')

That is, I iterated through all parameters, the key to failure is (of course) on GPU but also min_data_in_bin=1.
lgb257b.pkl.zip

Hi, any thoughts?  Seems like a clear MRE, but it's been 5 days and no response.  Thanks.

@guolinke ?

  File "/opt/h2oai/dai/python/lib/python3.8/site-packages/lightgbm_gpu/sklearn.py", line 712, in fit
    self._Booster = train(params, train_set,
  File "/opt/h2oai/dai/python/lib/python3.8/site-packages/lightgbm_gpu/engine.py", line 235, in train
    booster = Booster(params=params, train_set=train_set)
  File "/opt/h2oai/dai/python/lib/python3.8/site-packages/lightgbm_gpu/basic.py", line 2528, in __init__
    _safe_call(_LIB.LGBM_BoosterCreate(
  File "/opt/h2oai/dai/python/lib/python3.8/site-packages/lightgbm_gpu/basic.py", line 125, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: bin size 258 cannot run on GPU

Again,  no categorical handling enabled etc.

@guolinke reminder - still the dominant failure mode for LightGBM in Driverless AI

@pseudotensor The current CUDA version is doing the correct thing, it can handle categorical features normally. The only problem is current implementation only do histogram construction on GPU, so the GPU utilization can be low.
Supporting of categorical features is not added yet in our first part of new CUDA version #4630, but will be added later.

Here's another minimal repro, in case helps
lgb.bin257.pkl.zip
import pickle
import lightgbm as lgb
print(lgb.__version__)

from lightgbm.sklearn import LGBMRegressor
with open("lgb.bin257.pkl", "rb") as f:
    X, y = pickle.load(f)
    model = LGBMRegressor(max_bin=252, device_type='gpu')
    model.fit(X, y)
    print("OK1")

    model = LGBMRegressor(max_bin=253, device_type='gpu')
    model.fit(X, y)
    print("OK2")

first one passes, second one fails, not sure where 257 comes from:
/home/arno/minicondadai_py38/bin/python /nfs4/lgb_prefit_1c95733f-58d6-4a61-969f-b2331e03e895.py -m h2oai
<IP_ADDRESS>
OK1
[LightGBM] [Fatal] bin size 257 cannot run on GPU
Traceback (most recent call last):
  File "/nfs4/lgb_prefit_1c95733f-58d6-4a61-969f-b2331e03e895.py", line 13, in <module>
    model.fit(X, y)
  File "/home/arno/minicondadai_py38/lib/python3.8/site-packages/lightgbm/sklearn.py", line 851, in fit
    super().fit(X, y, sample_weight=sample_weight, init_score=init_score,
  File "/home/arno/minicondadai_py38/lib/python3.8/site-packages/lightgbm/sklearn.py", line 714, in fit
    self._Booster = train(params, train_set,
  File "/home/arno/minicondadai_py38/lib/python3.8/site-packages/lightgbm/engine.py", line 260, in train
    booster = Booster(params=params, train_set=train_set)
  File "/home/arno/minicondadai_py38/lib/python3.8/site-packages/lightgbm/basic.py", line 2537, in __init__
    _safe_call(_LIB.LGBM_BoosterCreate(
  File "/home/arno/minicondadai_py38/lib/python3.8/site-packages/lightgbm/basic.py", line 125, in _safe_call
    raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
lightgbm.basic.LightGBMError: bin size 257 cannot run on GPU

Process finished with exit code 1

Thanks very much @arnocandel !
But are you able to provide a reproducible example starting from raw data in a text-based format, generated from scratch with pandas / numpy / scipy code, or using a widely-distributed dataset like those available in sklearn.datasets?
I personally don't ever load pickle files whose origin I don't know, and I expect others wanting to contribute to fixing this issue might share that hesistation.
From https://docs.python.org/3/library/pickle.html

Warning The pickle module is not secure. Only unpickle data you trust.

It is possible to construct malicious pickle data which will execute arbitrary code during unpickling. Never unpickle data that could have come from an untrusted source, or that could have been tampered with.

X_y.zip

I'm having the same issue over here!
bin size 257 cannot run on GPU

@jameslamb - were you able to check with above two .csv files for X and y?

were you able to check with above two .csv files for X and y

I was not. If you're subscribed to this issue, you'll be notified when someone picks this up or has new information to share.

this is a bug for lightGBM for GPU,when use CPU,it is OK.

Any update so far on this issue?

I'm having the same issue :(

Still have this issue.

I have the same issue
"LightGBMError: bin size 1973 cannot run on GPU."
It runs alright using CPU.

For everyone who encounters this issue with the -DUSE_GPU=ON version of LightGBM, please check our latest GPU version which should be compiled with -DUSE_CUDA=ON.
https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-cuda-version. Thanks.

For everyone who encounters this issue with the -DUSE_GPU=ON version of LightGBM, please check our latest GPU version which should be compiled with -DUSE_CUDA=ON. https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-cuda-version. Thanks.

I have followed these instructions to install the CUDA version instead of the GPU version, but I still have the same issue:
LightGBMError: bin size XXX cannot run on GPU.
For more info, I am running on a linux server with cuda 12.1 with A100. Let me know if more info are needed to fix this issue.

Same issue, works fine on CPU
[LightGBM] [Fatal] bin size 260 cannot run on GPU

For everyone who encounters this issue with the -DUSE_GPU=ON version of LightGBM, please check our latest GPU version which should be compiled with -DUSE_CUDA=ON. https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-cuda-version. Thanks.

I have realized that after compiling lightgbm with the cuda option, and then using the command sudo sh ./build-python.sh install --precompile to install it as highlighted in the documentation, it defaults to installing the pip repo version. I have not verified that by inspecting the build-python.sh script, but  my workaround was to build the pip wheel package myself. This solves the issue, and when specifying device_type=cuda works correctly as expected.
On a side note, the main issue of cuda memory still persists, and this relates to the fact that a categorical feature has too many unique values (I tested by omitting that feature and it works fine on both gpu, cuda and cpu). But when including that feature, using the gpu version I get LightGBMError: bin size XXX cannot run on GPU, it works fine on the CPU, but takes a very long time, and using the cuda version, you can find the error below (optuna study multiple workers).
[LightGBM] [Fatal] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

[LightGBM] [Fatal] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/treelearner/cuda/cuda_best_split_finder.cu 2066

[LightGBM] [Fatal] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

[LightGBM] [Warning] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

[LightGBM] [Warning] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

[LightGBM] [Fatal] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

[LightGBM] [Fatal] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_tree.cpp 37

[LightGBM] [Warning] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

[LightGBM] [Fatal] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

terminate called after throwing an instance of 'std::runtime_error'
[LightGBM] [Warning] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

  what():  [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_tree.cpp 37

[LightGBM] [Fatal] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

[LightGBM] [Warning] [CUDA] an illegal memory access was encountered /home/user/notebooks/jupiter/tmp/src/io/cuda/cuda_column_data.cpp 67

So it seems that there is a limitation in the implementation when it comes to categorical features on cuda/gpu, that requires a fix.

I have the same issue
"LightGBMError: bin size 512 cannot run on GPU."


add way to add flexInts

since these are both a float64 and string representation, make it easier to keep these consistent. related to https://github.com/unpoller/dashboards/issues/20#issuecomment-1364716049

I'm also curious how this is getting used. These data structures are more of an output, and not really for manipulation, but maybe I forgot something I did haha. I don't see much about this particular need in the linked issue.

https://github.com/unpoller/unpoller/pull/503 this which is where I realized the Txt value was being used instead of the .Val for influx and prometheus.


k3s: v1.30.1+k3s1 -> v1.30.2+k3s2

Description of changes
Updating k3s (k3s_1_30) to v1.30.2.k3s2
Changelog: https://github.com/k3s-io/k3s/releases/tag/v1.30.2%2Bk3s2
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

k3s

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[X] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[X] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Is update script not working?

Is update script not working?

I used the update script for this PR, but build was failing on master until go 1.22.4 got merged from staging. I am sure the bot would have caught it eventually, but I figured I would create it since I was checking.

Is update script not working?

I used the update script for this PR, but build was failing on master until go 1.22.4 got merged from staging. I am sure the bot would have caught it eventually, but I figured I would create it since I was checking.

Whenever the bot is not proposing PRs, we have to check update script logs, like:

https://r.ryantm.com/log/k3s/2024-07-06.log

The commit message is missing from release:

Release: https://github.com/k3s-io/k3s/releases/tag/v1.30.2%2Bk3s1

I'll just run tests and we can have this merged shortly.

Result of nixpkgs-review pr 325894 run on aarch64-darwin 1

@NixOS/nixpkgs-merge-bot merge


string undefined

If I use: let variable name = " ",it will return undefined,The same is true of spliced strings

Hi, cannot reproduce. Please show more detailed example how i can reproduce this.

This is a website with this question https://hpcubes.com
I don't know why in the browser, a self-executing function and in Vue's methods, the string that defines the DOM tag becomes undefined
this is js,html,encode Configuration
test.zip


Add tweak32 parameter to musig_partial_sig_combine

With this PR it's possible to create MuSig signatures for public keys with pay to contract commitments:
secp256k1_musig_pubkey_combine(.., &P, pk_hash, pk, ...);
secp256k1_ec_pubkey_tweak_add(ctx, &Q, ec_commit_tweak);
/* Note same pk_hash as in pubkey combine but with tweaked key Q */
secp256k1_musig_session_initialize(..., &Q, pk_hash, ...);
...
secp256k1_musig_partial_sig_combine(..., ec_commit_tweak);

If the signers want to create a signature for the the normal combined key P they run the protocol just as before (i.e. initialize the session with P and give NULL to the partial_sig_combine tweak32 argument.

Rearranged privkey tweaking a small bit to reuse more code from eckey_privkey_mul. Squashed because I'm building a PR on top of this.

ACK.
This API is a little bit dangerous because it allows tweaking even without doing the p2c/taproot construction, but I think that's OK; it'd be a layer violation to put the specific p2c construction into secp-zkp.

This API is a little bit dangerous because it allows tweaking even without doing the p2c/taproot construction,

Hm, not sure if we can care about layer violations if this lib should remain high level.
I suggested a couple of weeks ago to add a taproot tweak function to libsecp upstream that does the right thing for normal/single taproot keys but Pieter argued that the existing tweak_add is sufficient.
Perhaps we could add an optional tweak_func argument to partial_sig_combine that by default is secure, i.e. hashes a prefix, key and tweak, with the prefix being something like "secp-zkp default musig tweak" and not the taproot prefix.

I think having a default that wasn't Taproot would be a bit silly. But yeah, I would be more comfortable with such an API.

On the other hand having a default that was Taproot is also potentially dangerous, if people were mistakenly adding Taproot spend conditions to their keys..

Next step would be #77 which has to move the tweak to pubkey_combine, but is still easy to misuse.


could you provide the demo2.mp4 in your code?

Hi,
cap = cv2.VideoCapture("demo2.mp4")
could you provide the demo2.mp4 in your code for test?

BAM!
https://user-images.githubusercontent.com/71572746/167343841-b5f427ae-6bf4-4bd0-aa49-dcac3634cc62.mp4
have fun!

i should probably add that


Fix ioc upload update behavior and change error response

Description
Fixes the ioc upload behavior when retaining old ioc indices. Previously used to delete ioc indices when source config gets indexed but now has the same behavior as s3 download where the background delete job will delete old ioc indices.
Also changes internal server errors to bad requests.
Issues Resolved
[List any issues this PR will resolve]
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Plz add tests


doc: Documentation typo in 'boards/weact/blackpill_f401cc/doc/index'

Links and text typo
In the section supported features, 'blackpill_f401ce' is mentioned instead of 'blackpill_f401cc'.
"The default configuration can be found in boards/weact/blackpill_f401ce/blackpill_f401ce_defconfig".
The above link should have been this link instead.
Environment

Page: boards/weact/blackpill_f401cc/doc/index
Version: main
SHA-1: f2c2cc334d486ec613463774ac009df3982b3221

@thetechmusician thanks for noticing it and opening an issue! Can you please send a pull request to fix it, by any chance?

@kartben added pull request for the fix #81291


Installation on Debian 10 fails with obscure SIGSEGV

Attempt to install on Debian stable (10) failed. I tried to install it a few months ago with the same result. Please advice how to proceed.
The log:
$ nim -v
Nim Compiler Version 1.4.8 [Linux: amd64]
...
$ nimble -v
nimble v0.13.1 compiled at 2021-07-12 15:07:02
$ nimble install --verbose gintro
nimble install --verbose gintro
    Reading official package list
Downloading https://github.com/stefansalewski/gintro using git
    Cloning latest tagged version: v0.9.3
 Attempting to execute hook installBefore in /tmp/nimble_16312/githubcom_stefansalewskigintro/gintro.nimble
preparing gintro
--2021-07-12 18:20:01--  https://raw.githubusercontent.com/StefanSalewski/oldgtk3/master/oldgtk3/gobject.nim
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 108015 (105K) [text/plain]
Saving to: ‘gobject.nim’

gobject.nim         100%[===================>] 105.48K  --.-KB/s    in 0.06s

2021-07-12 18:20:02 (1.86 MB/s) - ‘gobject.nim’ saved [108015/108015]

--2021-07-12 18:20:02--  https://raw.githubusercontent.com/StefanSalewski/oldgtk3/master/oldgtk3/glib.nim
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 256798 (251K) [text/plain]
Saving to: ‘glib.nim’

glib.nim            100%[===================>] 250.78K  --.-KB/s    in 0.06s

2021-07-12 18:20:02 (3.94 MB/s) - ‘glib.nim’ saved [256798/256798]

--2021-07-12 18:20:02--  https://raw.githubusercontent.com/StefanSalewski/oldgtk3/master/oldgtk3/gir.nim
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 33340 (33K) [text/plain]
Saving to: ‘gir.nim’

gir.nim             100%[===================>]  32.56K  --.-KB/s    in 0.006s

2021-07-12 18:20:02 (5.02 MB/s) - ‘gir.nim’ saved [33340/33340]

Hint: used config file '/home/amakarov/activity-shared/nim-stable/config/nim.cfg' [Conf]
Hint: used config file '/home/amakarov/activity-shared/nim-stable/config/config.nims' [Conf]
....................
/tmp/gintrosalewski/glib.nim(2914, 3) Hint: 'G_NORMALIZE_NFKC' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(3040, 3) Hint: 'G_IO_FLAGS_GET_MASK' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(31, 3) Hint: 'GTK_DOC_IGNORE' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(34, 3) Hint: 'INC_STDLIB' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(1806, 3) Hint: 'G_TRAVERSE_MASK' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(26, 3) Hint: 'MINGW_H' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(1807, 3) Hint: 'G_TRAVERSE_LEAFS' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(23, 3) Hint: 'GLIB_COMPILATION' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(2912, 3) Hint: 'G_NORMALIZE_NFC' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(1808, 3) Hint: 'G_TRAVERSE_NON_LEAFS' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(2911, 3) Hint: 'G_NORMALIZE_NFD' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(22, 3) Hint: 'GLIB_H_INSIDE' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(4652, 3) Hint: 'G_REGEX_MATCH_PARTIAL_SOFT' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(27, 3) Hint: 'STDLIB_H' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(2913, 3) Hint: 'G_NORMALIZE_NFKD' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/glib.nim(29, 3) Hint: 'MSC_VER' is declared but not used [XDeclaredButNotUsed]
.
/tmp/gintrosalewski/gobject.nim(658, 3) Hint: 'G_PARAM_READWRITE' is declared but not used [XDeclaredButNotUsed]
.........
/tmp/gintrosalewski/gen.nim(209, 6) Hint: 'endsWith' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/gen.nim(65, 10) Hint: 'isTypeInfo' is declared but not used [XDeclaredButNotUsed]
/tmp/gintrosalewski/gen.nim(261, 6) Hint: 'renumber' is declared but not used [XDeclaredButNotUsed]
Hint:  [Link]
Hint: 65450 lines; 1.138s; 117.824MiB peakmem; Debug build; proj: /tmp/gintrosalewski/gen.nim; out: /tmp/gintrosalewski/gen [SuccessX]
First we try generating bindings for GTK4, this may fail when GTK4 is not properly installed
on your computer. But don't worry, you can still use GTK3
Can't load gtk4 library!
Failed to load Gtk
Typelib file for namespace 'Gtk', version '4.0' not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
Failed to load Gdk
Typelib file for namespace 'Gdk', version '4.0' not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
Failed to load GdkX11
Typelib file for namespace 'GdkX11', version '4.0' not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
Failed to load Gsk
Typelib file for namespace 'Gsk' (any version) not found
Maybe for your OS you have to install additional GTK related packages?
We continue with the remaining packages...
graphene................: Remaining delayed methods: 0 (Fine!)
Traceback (most recent call last)
/tmp/gintrosalewski/gen.nim(4261) gen
/tmp/gintrosalewski/gen.nim(4192) launch
/tmp/gintrosalewski/gen.nim(3768) main
SIGSEGV: Illegal storage access. (Attempt to read from nil?)
stack trace: (most recent call last)
/tmp/nimblecache-3558437777/nimscriptapi_756804212.nim(199, 29)
/tmp/nimble_16312/githubcom_stefansalewskigintro/gintro.nimble(77, 7) installBefore
/tmp/nimble_16312/githubcom_stefansalewskigintro/gintro.nimble(64, 7) prep
/home/amakarov/activity-shared/nim-stable/lib/system/nimscript.nim(260, 7) exec
/home/amakarov/activity-shared/nim-stable/lib/system/nimscript.nim(260, 7) Error: unhandled exception: FAILED: /tmp/gintrosalewski/gen 1 [OSError]
     Error: Exception raised during nimble script execution

Thanks for reporting.
That is an interesting and strange bug, which has never been seen before. The error position gen.nim(3768) is really strange.

Are you using a 32 bit Linux or maybe a none x86 system like ARM or MIPS or RISCV.
I assume that you are very new to GTK, as you have never tried gintro in the last 6 years. So I would advice you to just use one of the 20 other Nim GUI toolkits. I have listed their names in the GTK4 book at the beginning, see  http://ssalewski.de/gtkprogramming.html. Araq and the other Nim core devs seems to advertise Fidget most currently. Learning GTK for beginners is a large task unfortunately, the gintro bindings are still not that good as the Rust bindings, and number of gintro users is small.
For you discovered bug, I will certainly try to discover the reason, but I have no good idea currently.

@StefanSalewski ,
No, it's just a normal x86_64 PC with 16 GB of RAM.

Araq and the other Nim core devs seems to advertise Fidget most currently

Thank you very much for advice, I will take a closer look at that library also.

We will close this issue now as we can currently not reproduce it and have now idea for the reason.


XPublish documentation link fails

The links to the documentation from the main page are not working: https://xpublish.readthedocs.io/en/0.0.1/
It looks like the version is not resolving correctly, because if I manually go to https://xpublish.readthedocs.io/en/0.3.3/getting-started/index.html then it works.

Fixed! Thanks for the report @jonmjoyce.


Updated bad link

Updated link from 'https://aks.ms/STP' to 'https://aka.ms/STP'
Module: 00
Lab/Demo: 00
Fixes # .
Changes proposed in this pull request:

Updated bad link from 'https://aks.ms/STP' to 'https://aka.ms/STP'

hi NathTheDude: thanks for calling this out, good catch.  I have updated the item and merged it into the code through a separate PR, so I am closing this one.  thanks again!


Verify NSNumber, CIColor and CIVector filters

[ ] Area Average			CIAreaAverage
[ ] Area Histogram			CIAreaHistogram
[ ] Area Logarithmic Histogram			CIAreaLogarithmicHistogram
[ ] Area Maximum			CIAreaMaximum
[ ] Area Maximum Alpha			CIAreaMaximumAlpha
[ ] Area Minimum			CIAreaMinimum
[ ] Area Minimum Alpha			CIAreaMinimumAlpha
[ ] Area Min and Max			CIAreaMinMax
[ ] Area Min and Max Red			CIAreaMinMaxRed
[ ] Bump Distortion			CIBumpDistortion
[ ] Bump Distortion Linear			CIBumpDistortionLinear
[ ] Checkerboard			CICheckerboardGenerator
[ ] Circle Splash Distortion			CICircleSplashDistortion
[ ] Circular Screen			CICircularScreen
[ ] Circular Wrap Distortion			CICircularWrap
[ ] Clamp			CIClamp
[ ] CMYK Halftone			CICMYKHalftone
[ ] Color Clamp			CIColorClamp
[ ] Color Cross Polynomial			CIColorCrossPolynomial
[ ] Color Matrix			CIColorMatrix
[ ] Color Polynomial			CIColorPolynomial
[ ] Column Average			CIColumnAverage
[ ] 3 by 3 Convolution			CIConvolution3X3
[ ] 5 by 5 Convolution			CIConvolution5X5
[ ] 7 by 7 Convolution			CIConvolution7X7
[ ] Horizontal 9 Convolution			CIConvolution9Horizontal
[ ] Vertical 9 Convolution			CIConvolution9Vertical
[ ] 3 by 3 RGB Convolution			CIConvolutionRGB3X3
[ ] 5 by 5 RGB Convolution			CIConvolutionRGB5X5
[ ] 7 by 7 RGB Convolution			CIConvolutionRGB7X7
[ ] Horizontal 9 RGB Convolution			CIConvolutionRGB9Horizontal
[ ] Vertical 9 RGB Convolution			CIConvolutionRGB9Vertical
[ ] Crop			CICrop
[ ] Crystallize			CICrystallize
[ ] Depth of Field			CIDepthOfField
[ ] Dot Screen			CIDotScreen
[ ] Droste			CIDroste
[ ] Eightfold Reflected Tile			CIEightfoldReflectedTile
[ ] Fourfold Reflected Tile			CIFourfoldReflectedTile
[ ] Fourfold Rotated Tile			CIFourfoldRotatedTile
[ ] Fourfold Translated Tile			CIFourfoldTranslatedTile
[ ] Gaussian Gradient			CIGaussianGradient
[ ] Glass Lozenge			CIGlassLozenge
[ ] Glide Reflected Tile			CIGlideReflectedTile
[ ] Hatched Screen			CIHatchedScreen
[ ] Hexagonal Pixelate			CIHexagonalPixellate
[ ] Hole Distortion			CIHoleDistortion
[ ] Kaleidoscope			CIKaleidoscope
[ ] Combined Keystone Correction			CIKeystoneCorrectionCombined
[ ] Horizontal Keystone Correction			CIKeystoneCorrectionHorizontal
[ ] Vertical Keystone Correction			CIKeystoneCorrectionVertical
[ ] Lenticular Halo			CILenticularHaloGenerator
[ ] Light Tunnel Distortion			CILightTunnel
[ ] Linear Gradient			CILinearGradient
[ ] Line Screen			CILineScreen
[ ] Nine Part Stretched			CINinePartStretched
[ ] Nine Part Tiled			CINinePartTiled
[ ] Op Tile			CIOpTile
[ ] Parallelogram Tile			CIParallelogramTile
[ ] Perspective Correction			CIPerspectiveCorrection
[ ] Perspective Tile			CIPerspectiveTile
[ ] Perspective Transform			CIPerspectiveTransform
[ ] Perspective Transform with Extent			CIPerspectiveTransformWithExtent
[ ] Pinch Distortion			CIPinchDistortion
[ ] Pixelate			CIPixellate
[ ] Pointillize			CIPointillize
[ ] Radial Gradient			CIRadialGradient
[ ] Rounded Rectangle Generator			CIRoundedRectangleGenerator
[ ] Row Average			CIRowAverage
[ ] Sixfold Reflected Tile			CISixfoldReflectedTile
[ ] Sixfold Rotated Tile			CISixfoldRotatedTile
[ ] Smooth Linear Gradient			CISmoothLinearGradient
[ ] Spot Light			CISpotLight
[ ] Star Shine			CIStarShineGenerator
[ ] Stretch Crop			CIStretchCrop
[ ] Stripes			CIStripesGenerator
[ ] Sunbeams			CISunbeamsGenerator
[ ] Temperature and Tint			CITemperatureAndTint
[ ] Tone Curve			CIToneCurve
[ ] Torus Lens Distortion			CITorusLensDistortion
[ ] Triangle Kaleidoscope			CITriangleKaleidoscope
[ ] Triangle Tile			CITriangleTile
[ ] Twelvefold Reflected Tile			CITwelvefoldReflectedTile
[ ] Twirl Distortion			CITwirlDistortion
[ ] Vignette Effect			CIVignetteEffect
[ ] Vortex Distortion			CIVortexDistortion
[ ] Zoom Blur			CIZoomBlur

Rect.fromLTWH(0, 0, 640, 480); ----> CIVector(cgRect: CGRect(x: 0, y: 0, width: 640, height: 480))
Point( 640, 480); ----> CIVector(cgPoint: CGPoint(x: 640, y: 480))
[1, 2, 3, 4, 5, 6] ----> CIVector(values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0], count: 7)

Area Average CIAreaAverage,  Area Histogram CIAreaHistogram, Area Logarithmic Histogram CIAreaLogarithmicHistogram,  Area Maximum CIAreaMaximum, Area Maximum Alpha CIAreaMaximumAlpha, Area Minimum CIAreaMinimum, Area Minimum Alpha CIAreaMinimumAlpha, Area Min and Max CIAreaMinMax, Area Min and Max Red CIAreaMinMaxRed, Column Average CIColumnAverage, Row Average CIRowAverage - this screens has two small pink screens instead of photos - on bottom and top screen, and nothing on the center screen;
Area Logarithmic Histogram CIAreaLogarithmicHistogram - can't enter _scale parameter;
Clamp CIClamp - blurry turquoise screens instead of photos;
Color Clamp CIColorClamp, 3 by 3 Convolution CIConvolution3X3 - too light;
Circular Wrap Distortion CICircularWrap - too big radius and pink fone;
Circle Splash Distortion CICircleSplashDistortion, Checkerboard CICheckerboardGenerator, Bump Distortion CIBumpDistortion, Droste CIDroste, Eightfold Reflected Tile CIEightfoldReflectedTile,  Fourfold Reflected Tile CIFourfoldReflectedTile,  Fourfold Rotated Tile CIFourfoldRotatedTile, Fourfold Translated Tile CIFourfoldTranslatedTile, Gaussian Gradient CIGaussianGradient,  Glide Reflected Tile CIGlideReflectedTile, Light Tunnel Distortion CILightTunnel, Linear Gradient CILinearGradient, Op Tile CIOpTile, Parallelogram Tile CIParallelogramTile, Perspective Tile CIPerspectiveTile, Radial Gradient CIRadialGradient, Rounded Rectangle Generator CIRoundedRectangleGenerator, Sixfold Reflected Tile CISixfoldReflectedTile, Sixfold Rotated Tile CISixfoldRotatedTile, Smooth Linear Gradient CISmoothLinearGradient, Spot Light CISpotLight, Star Shine CIStarShineGenerator, Stripes CIStripesGenerator, Sunbeams CISunbeamsGenerator, Triangle Kaleidoscope CITriangleKaleidoscope, Triangle Tile CITriangleTile, Twelvefold Reflected Tile CITwelvefoldReflectedTile, Zoom Blur CIZoomBlur - can't open this filters in Xcode (but majority open in flutter simulator (nothing to compare));
Color Cross Polynomial CIColorCrossPolynomial, Color Matrix CIColorMatrix, Color Polynomial CIColorPolynomial, Lenticular Halo CILenticularHaloGenerator - only standard values, do not work for other preset values;
5 by 5 RGB Convolution CIConvolutionRGB5X5 - can't open this filter in flutter simulator;
Horizontal 9 RGB Convolution CIConvolutionRGB9Horizontal, 7 by 7 RGB Convolution CIConvolutionRGB7X7, 3 by 3 Convolution CIConvolution3X3, Vertical 9 RGB Convolution CIConvolutionRGB9Vertical -  when you change _bias parameter - white screens;
Glass Lozenge CIGlassLozenge, Hole Distortion CIHoleDistortion, Kaleidoscope CIKaleidoscope, Perspective Correction CIPerspectiveCorrection, Perspective Transform CIPerspectiveTransform, Perspective Transform with Extent CIPerspectiveTransformWithExtent, Pinch Distortion CIPinchDistortion, Stretch Crop CIStretchCrop - very big different;
Combined Keystone Correction CIKeystoneCorrectionCombined,  Vertical Keystone Correction CIKeystoneCorrectionVertical - yes, but this screens has pink fone on bottom and top screen;
Twirl Distortion CITwirlDistortion - too big radius

Area Average CIAreaAverage, Area Histogram CIAreaHistogram,  Area Logarithmic Histogram CIAreaLogarithmicHistogram, Area Maximum CIAreaMaximum, Area Minimum Alpha CIAreaMinimumAlpha, Area Min and Max CIAreaMinMax,  Area Min and Max Red CIAreaMinMaxRed, Column Average CIColumnAverage, Row Average CIRowAverage - don't working, this screens has only two small pink screens instead of photos - on bottom and top screen, and nothing on the center screen;
Area Logarithmic Histogram CIAreaLogarithmicHistogram - working in flutter, but can't enter _scale parameter;
Checkerboard CICheckerboardGenerator, Gaussian Gradient CIGaussianGradient, Lenticular Halo CILenticularHaloGenerator (this filter working with only constant values), Radial Gradient CIRadialGradient, Rounded Rectangle Generator CIRoundedRectangleGenerator, Smooth Linear Gradient CISmoothLinearGradient, Star Shine CIStarShineGenerator, Stripes CIStripesGenerator, Sunbeams CISunbeamsGenerator - working in flutter, but error in Xcode(Thread 1: "[<CICheckerboardGenerator 0x600000b40800> setValue:forUndefinedKey:]: this class is not key value coding-compliant for the key inputImage.");
Circular Wrap Distortion CICircularWrap, Color Matrix CIColorMatrix, Glass Lozenge CIGlassLozenge, Hole Distortion CIHoleDistortion, Kaleidoscope CIKaleidoscope, Perspective Transform CIPerspectiveTransform, Perspective Transform with Extent CIPerspectiveTransformWithExtent, Pinch Distortion CIPinchDistortion, Twirl Distortion CITwirlDistortion - working in flutter, but very big different;
Clamp CIClamp - only blue screen;
Color Clamp CIColorClamp - yes, but you can't enter another _minComponents parameter (in Xcode too);
3 by 3 Convolution CIConvolution3X3 , 5 by 5 RGB Convolution CIConvolutionRGB5X5 , 7 by 7 RGB Convolution CIConvolutionRGB7X7, Horizontal 9 Convolution CIConvolution9Horizontal, Vertical 9 RGB Convolution CIConvolutionRGB9Vertical - working in flutter, but you can't center another _bias parameter (in Xcode too);
Color Polynomial CIColorPolynomial - working in flutter, but needs more saturation;
Droste CIDroste, Eightfold Reflected Tile CIEightfoldReflectedTile, Fourfold Reflected Tile CIFourfoldReflectedTile, Fourfold Rotated Tile CIFourfoldRotatedTile, Fourfold Translated Tile CIFourfoldTranslatedTile, Glide Reflected Tile CIGlideReflectedTile, Light Tunnel Distortion CILightTunnel, Linear Gradient CILinearGradient, Op Tile CIOpTile, Parallelogram Tile CIParallelogramTile, Perspective Tile CIPerspectiveTile, Sixfold Reflected Tile CISixfoldReflectedTile, Sixfold Rotated Tile CISixfoldRotatedTile, Triangle Kaleidoscope CITriangleKaleidoscope, Triangle Tile CITriangleTile, Twelvefold Reflected Tile CITwelvefoldReflectedTile - working in flutter, but can't open i Xcode for comparison;
Vertical Keystone Correction CIKeystoneCorrectionVertical, Combined Keystone Correction CIKeystoneCorrectionCombined, Perspective Correction CIPerspectiveCorrection, Perspective Transform with Extent CIPerspectiveTransformWithExtent, Rounded Rectangle Generator CIRoundedRectangleGenerator, Stretch Crop CIStretchCrop - working in flutter, but modified photos are located on the background of pink screens, there are pink backgrounds at the top and bottom of the screen, they are absent in the center;
Gaussian Gradient CIGaussianGradient - don't working (only white screen)


Convert html to react using react bootstrap => index.html

Index.html -> HomePage.jsx
header -> header.jsx

index page Done -> https://github.com/prasadhonrao/devcamper/pull/50

This issue was closed by PR #50


Cipher 0x001E mapping

Currently it points to TLS_FORTEZZA_KEA_WITH_RC4_128_SHA which is standardised in RFC6101. However according to https://www.ietf.org/mail-archive/web/tls/current/msg00036.html it is actually TLS_KRB5_WITH_DES_CBC_SHA.
Searching around openssl, this was previously brought up in #1743
This conflict is notable for me because TLS_KRB5_WITH_DES_CBC_SHA is on the http2 banned ciphers list (see RFC7540 appendix A)

The constant dates to 93ab9e421e4517b81d11ca741d55e6bb6f696c40. Even then, the only other reference was in the documentation:

SSL_FORTEZZA_KEA_WITH_RC4_128_SHA       Not implemented.

Interestingly, OpenSSL believes 0x1D and 0x1E are the Fortezza ciphers, while the IANA registry says:

0x00,0x1C-1D | Reserved to avoid conflicts with SSLv3

This comes from RFC 5246:

Note: The cipher suite values { 0x00, 0x1C } and { 0x00, 0x1D } are
reserved to avoid collision with Fortezza-based cipher suites in
SSL 3.

However, in RFC 6101, it says the Fortezza ciphers are:

CipherSuite SSL_FORTEZZA_KEA_WITH_NULL_SHA         = { 0X00,0X1C };
CipherSuite SSL_FORTEZZA_KEA_WITH_FORTEZZA_CBC_SHA = { 0x00,0x1D };
CipherSuite SSL_FORTEZZA_KEA_WITH_RC4_128_SHA      = { 0x00,0x1E };

It seems SSL 3.0 originally had three Fortezza ciphers, 0x1C through 0x1E, but this was mistranscribed into TLS as only 0x1C and 0x1D, and then TLS_KRB5_WITH_DES_CBC_SHA collided with SSL_FORTEZZA_KEA_WITH_RC4_128_SHA.
I also found this thread here, which says:
https://bugzilla.mozilla.org/show_bug.cgi?id=61932

I'd spotted the conflict. There was some discussion of this on ietf-tls
recently, where there was a suggestion that the single DES based 0x1e
cipher suite should just not be implemented. This is the route I'm currently
taking.

At this point, both ciphers are legacy, so which is the "actual" name is kind of academic. :-)

See also the recent discussion at https://www.ietf.org/mail-archive/web/tls/current/msg26454.html

Can it be confirmed that OpenSSL has never, and never will, implement either of FORTEZZA_KEA_WITH_RC4_128_SHA  or KRB5_WITH_DES_CBC_SHA?

We never had Fortezza code. We did have some Kerberos code but I don’t recall what it did. We are highly unlikely to implement those old, weak, security algorithms but there are very few promises about the future that we can actually make.

Given the Kerberos cipher uses single-DES which costs about $30 to break, it seems like a fairly safe bet though. :-)

We did have some Kerberos code but I don’t recall what it did.

there are patches to make if fully functional, but yes, given that it uses single DES, it should not be used at all

there are patches to make if fully functional

If there were patches for Kerberos but not for Fortezza, then we should give the id to the Kerberos cipher.

https://github.com/wahern/luaossl/pull/142/commits/e75a942f1b9783230d06235589b5f1413f05f359 is pending merge if the above sounds reasonable.

This doesn't seem worth doing.  Those code-points have a mangled history, and aren't currently implemented.

okay.... considering that history I think I'm going to merge my commit to the luaossl project anyway: there's always the chance that someone out there is using an openssl < 1.1.1 with the kerberos code around.

This is completely obsolete now. Closing.


support k8s 1.30 and 1.31

There are no changes  that will cause StatafulSet to restart since latest release. So I reverted update of moco-agent.


New version of extension not visible in VS Code UI

I published a new version of the InterSystems Language Server extension this morning (May 23rd). I published using vsce and use the platform-specific vsix feature. I can see the new version (2.0.3) in the marketplace listing UI in my browser:

However, within VS Code (stable and insiders on Intel Mac), version 2.0.2 is shown as the most recent version:

This is possibly related to https://github.com/microsoft/vscode/issues/147801

pinging @prashantvc

Update: I can see version 2.0.3 in the VS Code UI now and was prompted to update.

@isc-bsaviano Thanks for confirmation
@prashantvc I suspect this is because of some delay in Marketplace server/cache updates?


mmap module for windows

we added linux mmap in https://github.com/RustPython/RustPython/pull/3755
adding initializer and a few tuning will let windows also run the module.

I would like to claim this issue. I'm currently reading through the relevant python docs, linked threads, and the src. If anyone has any suggestions at where to direct my attention, it would be greatly appreciated.

Hello, I have rough steps but I don't have detailed descriptions.

Try to change memmap2 stdlib/Cargo.toml
The major difference of windows version is constructor. We can start with constructor and try to solve other stuff later.
Run test_mmap to check how it is going. cargo run Lib/test/test_mmap.py. You don't need to make everything working.

@killme2008 do you have any advice?

@youknowone  I don't have more advice, it's a good guide to start.


chore: FileValidation error vs Server error

Summary | Résumé
Fixes #4478
A previous fix (#4374) caused all errors from the serverAction to be caught as FileValidation errors.
This will specifically catch those and render appropriately, and let everything else fall through to a generic Server error.

Tested on preview env.
Working as expected.


Add capability to specify which hosts a PSProfile setting should be applicable to

It would be useful to be able to specify what PowerShell hosts (PowerShell Core, Windows PowerShell, PowerShell ISE, VS Code, etc) a specific init script, script, module and so on should be loaded in. Default value could be "All" for such a setting.
This would make it possible to only load specific things in a specific host, for example some modules don't work in all hosts.
Another use case is that one might not want "heavy" things like large modules and such loaded when working in a specific environment such as VS Code, where a fast startup time is convenient when testing different things, reloading the integrated console and so on.

I like it! I've been thinking about a way to add conditionals to each setting, which I think would cover this in a way. My initial thought was to support a ScriptBlock condition property on each non-essential section/setting where it makes sense. Thoughts?

Yes, that would make it really flexible with regards to other conditions as well. Such as hostname (useful when sharing the same config across multiple machines) and operating systems.
Thumbs up 👍


Added cni plugin

Updated the documentation
The link to install CNI is not valid
https://projectcalico.docs.tigera.io/manifests/calico.yaml so updated with
KUBECONFIG=capc-cluster.kubeconfig kubectl apply -f https://raw.githubusercontent.com/weaveworks/weave/master/prog/weave-kube/weave-daemonset-k8s-1.11.yaml

@kiranchavala you'll need to signup for the EasyCLA (you may click on the link and agree your contributions as an individual)
/approve

@kiranchavala
Have you tested weavenet ?
I used calico this April, it worked well, except the wrong URL which should be https://raw.githubusercontent.com/projectcalico/calico/master/manifests/calico.yaml

@weizhouapache tested weave net cni and  its working fine
Calico is also by me if its pointing to a valid URL

@weizhouapache
Add command for both calico and weave net

/ok-to-test
/check-cla

/assign @weizhouapache
Wei please review and lgtm to merge if you agree with changes, thanks.

/check-cla

@kiranchavala
could you sign the CLA ?

/lgtm


Search doesn't consider nodes of unloaded topologies

When doing a generic query (landing on the PODS topology), we only get that topology matched by the query:

Then when switching to a different topology, both of them are matched:

This seems to imply that only the loaded nodes are matched, as opposed to all the nodes across all topologies, as it should be by design of getAllNodes method.
Note: This only happens on dev and cloud for some reason, not locally for some reason.

Fixed by #2395.


B2B-1281 upgrade to link token usage

fixes: https://front-finance.atlassian.net/browse/B2B-1281

@afzalahmad-front Seems like this change would not maintain backwards compatibility, since the url parameter is taking the linkToken value.
We should implement this so, that both of them can be used, but the url property is marked as obsolete.


-Dexec.garden breaks script execution with trailing slash

Having a trailing slash on the -Dexec.garden parameter makes it impossible to execute scripts.
Log:

12-09-2017 16:17:42.873
WARN
Could not execute request: /usr/local/HaBridge/bin//ha-105.sh <IP_ADDRESS> off with message: Cannot run program "/usr/local/HaBridge/bin//ha-105.sh": error=2, No such file or directory
com.bwssystems.HABridge.plugins.exec.CommandHome

Jar execution:
/usr/local/bin/java -Djava.net.preferIPv4Stack=true -Dconfig.file=/usr/local
/HaBridge/data/habridge.config -Dexec.garden=/usr/local/HaBridge/bin/ -jar /usr/l
ocal/HaBridge/ha-bridge-5.0.0.jar
Note the trailing slash on the /bin/ folder.
The script execute command:
ha-105.sh <IP_ADDRESS> off
It appears the program is prepending the script execution with a / (it does this using exec://ha-105.sh as well) and then joining that (strings?) to the garden path. Java should be able to resolve this path more robustly.
It looks like the code doing this is at line 47 of ComandHome.java:
		if(execGarden != null && !execGarden.trim().isEmpty()) {
			if(System.getProperty("os.name").toLowerCase().indexOf("win") >= 0)
				intermediate = execGarden + "\\" + intermediate;
			else
				intermediate = execGarden + "/" + intermediate;
		}

You can skip this custom logic by just using java File objects (or Paths, but those are needlessly complex for this), like this:
		if(execGarden != null && !execGarden.trim().isEmpty()) {
			intermediate = new File(execGarden, intermediate).getAbsolutePath();
		}

Much appreciated. Always good to have a second set of eyes help!


Customize loading and error styles

At present, I can only modify the styles of loading and error, but I want to customize the content of loading and error, such as adding the text "Thinking about it..." to loading, and modifying the text "Something wet wrong.. Please try again" to error. How should it be implemented?

It would be even better if the input box could display "current input characters/total number of characters that can be inputted", such as "20/100", to limit the number of characters that can be inputted through "characterLimit"!

Hi @TasonIV.
Modifying the loading bubble's content and adding a character input number tracker are features that we will be adding in the future. We currently have multiple higher priority features that we are working on, hence it will take some time before we can work on the aforementioned ones.
Ofcourse, you are welcome to fork/clone the project and modify it as needed. It is very easy to do and the setup steps are documented in create your own Deep Chat component section.
If you want to control the loading message bubble now, you can alternatively use the handler function like a websocket (check the Websocket tab). This will allow you to create a fake loading message via the overwrite property whilst you call your service. Example. Ofcourse the only downside is that you have to write your own fetch logic, but it is not hard to do using this fetch webpage.
Your code would be something like the following:
chatElementRef.request = {
  websocket: true,
  handler: (_, signals) => {
    signals.onOpen(); // enables the user to send messages
    // triggered when the user sends a message
    signals.newUserMessage.listener = async (body) => {
      try {
        signals.onResponse({text: 'Thinking about it...'});
        // await fetch request to your api using the body
        setTimeout(() => signals.onResponse({text: 'Actual result', overwrite: true}), 1000);
      } catch (e) {
        signals.onResponse({error: 'error'}); // displays an error message
      }
    };
  },
};

I will be closing this issue. If you have any further questions feel free to comment below, for anything else you are more than welcome to create a new issue. Thanks!


setup.py is missing out package 'admin'

I've installed papermerge through setup.py but the resulting package is missing out 'admin' and isn't starting.
A workaround to for this is to copy papermerge/contrib into build/lib/papermerge before running python setup.py build and python setup.py install --root="$pkgdir" --optimize=1 --skip-build later.

Hi @Lucki,
thank you for trying Papermerge.
In theory it sounded to me as if it is a great idea to have a pypi package for papermerge.
But in practice - I never use papermerge pypi package.
I always end-up cloning from the repo - even in production environments.
This explains why "admin" is missing (papermerge.contrib.admin was introduced since about 2 months ago).
Thank you for bringing up this issue, I will take care of it!

I actually did create all necessary PKGBUILDs for Arch Linux but currently I'm not happy with Papermerge after some fiddling around so I think I'm not going to push them into the AUR.

I'm not happy with Papermerge after some fiddling around

Maybe you can list points which you didn't like in Papermerge here. I would gladly consider your feedback and improve application in future.

I actually did create all necessary PKGBUILDs for Arch Linux but currently I'm not happy with Papermerge after some fiddling around so I think I'm not going to push them into the AUR.

I was just looking for a nice PKGBUILD in the AUR, so I can only encourage you to publish it there and maybe really state the issues you are having. Sometimes, solutions can be found unexpectedly easily ;)

Maybe you can list points which you didn't like about Papermerge here.

I'll do but I need to figure out the exact points first. In general it's that the whole usage feels really clunky and I doubt they aren't already mentioned somewhere around here.

I was just looking for a nice PKGBUILD in the AUR, so I can only encourage you to publish it there

I don't want to maintain these without using them so I won't publish them there. But I drop them here in case someone wants to pick them up.
The main PKGBUILD with the associated sysusers, service unit and tmpfiles needs some general polish. I was in the process of hooking gunicorn between django and nginx but my OCR broke while doing that. That's the current state.

papermerge
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

pkgname=papermerge
pkgver=1.4.3
pkgrel=2
pkgdesc="An open source document management system (DMS) primarily designed for archiving and retrieving your digital documents."
arch=('any')
url="https://github.com/ciur/papermerge"
license=('Apache')
backup=("etc/$pkgname.conf.py" "usr/lib/python3.8/site-packages/config/settings/production.py")
depends=('imagemagick' 'pdftk' 'poppler' 'python-celery' 'python-django' 'python-django-allauth' 'python-django-celery-results' 'python-django-dynamic-preferences'
      'python-django-modelcluster' 'python-django-mptt' 'python-django-polymorphic-tree' 'python-django-rest-framework' 'python-django-rest-knox' 'python-django-taggit'
      'python-imapclient' 'python-lxml' 'python-mgclipboard' 'python-mglib' 'python-psycopg2' 'python-pyaml' 'tesseract')
optdepends=('gunicorn: run papermerge in a production environment with nginx'
          'nginx: run papermerge through gunicorn in a production environment')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/ciur/$pkgname/archive/v$pkgver.tar.gz"
      "$pkgname.sysusers"
      "$pkgname.tmpfiles"
      "$pkgname-gunicorn.service"
      "$pkgname-worker.service"
      "$pkgname.sh")
sha512sums=('1665873c666424c456655f91a5e980a519945d937c2ae351306add4b44d3618e4a8700f20d08fb52639c1b37b90f1dfe4565dcfbb8452897133880d83daeb1ea'
          'eb408d6d6c0072b4883cae9b6aa591935b910585e93aa6fdacbb543dbc4233588678d396c06d0f10b7c74b4d393af19b8be526d094821de23792c29df593d4a5'
          '07726d47deb2438dbeb3f5dc6ebc2218a4d5f25dacb7e0304102deddfb914d4d6d649643e26cb844ac4d02c7f21b58835edd4026ff91a6481fee59f786b90925'
          'c99103b8a160dde851e169d4249da8ade66377f428df0784401309a1c181ce4c5011c31b28d26981456e4490f670ff161bf0ebc0c9daf47cdf761b4d5cb9cb32'
          'fcdd49e0b6b93ade84ddafd77111f28bdea75a4f12829ec249d414d46a14399198be7bf29d0ce5166de6dba1d6386eda4ab9b39a1f5376b7c7758c8966a53ce3'
          'b838d54dca56e45f99690ddaae1b058e3f72b314adf9d689869f14bb141a5bbf209df974eea20cb2305b127d5119d0b8d11293494d47350f6974edc6fffc374f')

prepare() {
  {
      echo 'DBDIR = "/var/lib/papermerge/database"';
      echo 'MEDIA_DIR = "/var/lib/papermerge/media"';
      echo 'STATIC_DIR = "/var/lib/papermerge/static"';
      echo 'TASK_QUEUE_DIR = "/var/tmp/papermerge/queue"'
  } >> "$pkgname-$pkgver/$pkgname.conf.py.example"

  # https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html#step-1-install-gunicorn
  {
      echo 'from .base import *  # noqa';
      echo 'DEBUG = False';
      echo "ALLOWED_HOSTS = ['*']"
  } > "$pkgname-$pkgver/config/settings/production.py"

  # The admin package is missing somehow so let's move it plain stupid directly in place
  mkdir -p "$pkgname-$pkgver/build/lib/$pkgname/"
  cp -dpr --no-preserve=ownership "$pkgname-$pkgver/$pkgname/contrib" "$pkgname-$pkgver/build/lib/$pkgname/"
}

build() {
  cd "$pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$pkgname-$pkgver" || exit
  ./run_tests.sh
}

package() {
  # tmpfiles.d
  install -Dm 644 "$pkgname.tmpfiles" \
      "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf"

  # sysusers.d
  install -Dm 644 "$pkgname.sysusers" \
      "$pkgdir/usr/lib/sysusers.d/$pkgname.conf"

  # service files
  install -Dm 644 "$pkgname-gunicorn.service" \
      "$pkgdir/usr/lib/systemd/system/$pkgname-gunicorn.service"
  install -Dm 644 "$pkgname-worker.service" \
      "$pkgdir/usr/lib/systemd/system/$pkgname-worker.service"

  # wrapper
  install -Dm 755 "$pkgname.sh" \
      "$pkgdir/usr/bin/$pkgname-manage.py"

  cd "$pkgname-$pkgver" || exit

  # config
  install -Dm 644 "$pkgname.conf.py.example" \
      "$pkgdir/etc/$pkgname.conf.py"

  export PYTHONHASHSEED=0
  python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}

papermerge-worker.service
[Unit]
Description=Papermerge Worker
Documentation=https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html
After=network.target
PartOf=papermerge-gunicorn.service

[Service]
Type=simple
User=papermerge
Group=papermerge
WorkingDirectory=/tmp
Environment="DJANGO_SETTINGS_MODULE=config.settings.production"
ExecStart=/usr/bin/django-admin worker --pidfile /var/run/papermerge/papermerge-worker.pid
Restart=on-failure

[Install]
WantedBy=multi-user.target

papermerge-gunicorn.service
[Unit]
Description=Gunicorn providing papermerge
Documentation=https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html,https://docs.gunicorn.org/en/latest/deploy.html#systemd
After=network.target
Wants=papermerge-worker.service

[Service]
Type=notify
User=papermerge
Group=papermerge
RuntimeDirectory=papermerge
WorkingDirectory=/var/lib/papermerge
Environment="DJANGO_SETTINGS_MODULE=config.settings.production"
ExecStart=/usr/bin/gunicorn config.wsgi:application --config /etc/gunicorn.conf.py
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true
Restart=on-failure

[Install]
WantedBy=multi-user.target

papermerge.sh
#!/bin/sh
export DJANGO_SETTINGS_MODULE=config.settings.dev
cd /tmp || exit
exec /usr/bin/django-admin "$@"

papermerge.sysusers
u papermerge - "open source document management system" -

papermerge.tmpfiles
d /var/lib/papermerge/media - papermerge papermerge -
d /var/lib/papermerge/static - papermerge papermerge -
d /var/lib/papermerge/database - papermerge papermerge -
d /run/papermerge - papermerge papermerge -
d /var/log/papermerge - papermerge papermerge -

python-django-celery-results
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-celery-results
pkgname=python-django-celery-results
pkgver=1.2.1
pkgrel=1
pkgdesc="Store Celery task results using the Django ORM."
arch=('any')
url="https://github.com/celery/django-celery-results"
license=('BSD')
depends=('python-celery')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/celery/django-celery-results/archive/$pkgver.tar.gz")
sha512sums=('9f25d8ecf9716dca04be36f608d682ec138f0d6ceeaff2acdbe3f2c617e0eab8b0f8748dd35eead9fd7c40cdfbfd2ed9663a586a41f70f3a4bebecd3aba90d3d')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# check() {
#     # TODO
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}

python-django-modelcluster
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-modelcluster
pkgname=python-django-modelcluster
pkgver=5.1
pkgrel=1
pkgdesc="Django extension to allow working with 'clusters' of models as a single unit, independently of the database."
arch=('any')
url="https://github.com/wagtail/django-modelcluster"
license=('BSD')
depends=('python-pytz')
optdepends=('python-django-taggit')
makedepends=('python-setuptools')
checkdepends=('python-django' 'python-django-taggit')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/wagtail/django-modelcluster/archive/v$pkgver.tar.gz")
sha512sums=('bb3d8640a04cabfdebe7f60b571a2b2c4d71bbf5d4ea0ff0e7db5fe9d120e6eab7ec3f1b6992d21aff377045f8ed3e11d1b351ce2f460b33dd4c0f3a8dd749fb')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$_pkgname-$pkgver" || exit
  python runtests.py
}

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}

python-django-mptt
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-mptt
pkgname=python-django-mptt
pkgver=0.11.0
pkgrel=1
pkgdesc="Utilities for implementing a modified pre-order traversal tree in django."
arch=('any')
url="https://github.com/django-mptt/django-mptt"
license=('custom')
depends=('python-django' 'python-django-js-asset')
makedepends=('python-setuptools')
checkdepends=('python-coverage' 'python-mock')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/django-mptt/django-mptt/archive/$pkgver.tar.gz")
sha512sums=('0302fbdc759fe3e69c7adc3e785025c4c28931904c2b8904d43a3fdf1471ef1a83f18170f1db05dc3d4360e946c35db06900cf9c40b87562fba03240f9608da7')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# TODO: AssertionError: False is not true : Couldn't find 'javascript" src="/static/mptt/draggable-admin.js" data-context="{"' in response
# check() {
#     cd "$_pkgname-$pkgver/tests" || exit
#     ./runtests.sh
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}

python-django-polymorphic-tree
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-polymorphic-tree
pkgname=python-django-polymorphic-tree
pkgver=1.5.1
pkgrel=1
pkgdesc="Polymorphic MPTT tree support for models"
arch=('any')
url="https://github.com/django-polymorphic/django-polymorphic-tree"
license=('Apache')
depends=('python-django' 'python-django-mptt' 'python-django-polymorphic' 'python-django-tag-parser' 'python-future')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/django-polymorphic/django-polymorphic-tree/archive/v$pkgver.tar.gz")
sha512sums=('291ed73400c3a5073cbfe00fdd0dab16ac7dbb4fa96fc1797f526ef7235111162806c4be3fddb0bc6cd61560bdb6d1ac806c2f83380a1d45735d876c4f79ac67')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$_pkgname-$pkgver" || exit
  python runtests.py
}

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"
}

python-django-rest-knox
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-rest-knox
pkgname=python-django-rest-knox
pkgver=4.1.0
pkgrel=1
pkgdesc="Authentication Module for django rest auth"
arch=('any')
url="https://github.com/James1345/django-rest-knox"
license=('MIT')
depends=('python-cryptography' 'python-django' 'python-django-rest-framework')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/James1345/django-rest-knox/archive/$pkgver.tar.gz")
sha512sums=('ff7252e35ed3998d882ba5f857ba7278340065ab849475b0b3d46a04842eac43f38f089fc138ebbe1495a06c4fd4ddb7bd1025a816ae4ad2c646752e74f85336')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# check() {
#     https://wiki.archlinux.org/index.php/Python_package_guidelines#Check
#     > Avoid using tox to run testsuites
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"

  install -Dm644 LICENSE -t "$pkgdir/usr/share/licenses/$pkgname"
}

python-django-tag-parser
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

_pkgname=django-tag-parser
pkgname=python-django-tag-parser
pkgver=3.2
pkgrel=1
pkgdesc="Micro-library to easily write custom Django template tags "
arch=('any')
url="https://github.com/edoburu/django-tag-parser"
license=('Apache')
depends=('python')
makedepends=('python-setuptools')
checkdepends=('python-django')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/edoburu/django-tag-parser/archive/v$pkgver.tar.gz")
sha512sums=('ab39dcf847b1da473ded83655bd8ba7a6e376de68668cd9dc7b2019d8666864ed5665ab162f4a7df56cb7a3eb32e83c8fad3fdecf5774140effc643853997879')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$_pkgname-$pkgver" || exit
  python runtests.py
}

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --skip-build \
      --optimize=1 \
      --prefix=/usr \
      --root="$pkgdir"
}

python-mgclipboard
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

pkgname=python-mgclipboard
_pkgname=mg-clipboard
pkgver=0.3.0
pkgrel=1
pkgdesc="Modules shared across all Papermerge project."
arch=('any')
url="https://github.com/papermerge/mg-clipboard"
license=('Apache')
depends=('python')
makedepends=('python-setuptools')
_commit="f5a4e9370820c3d8bbfbc3bed08131d32bc141fc"
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/papermerge/mg-clipboard/archive/$_commit.tar.gz")
sha512sums=('25f7787ca3aec9d0f56ddf4dc65a530a39c4238a91040af592b8f5c686f539ddcf734101307aa9ee86f6850825c7eb33ea66e2cfd19c27a96358e755bf1b78a9')

build() {
  cd "$_pkgname-$_commit" || exit
  python setup.py build
}

package() {
  cd "$_pkgname-$_commit" || exit

  export PYTHONHASHSEED=0
  python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}

python-mglib
# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148

pkgname=python-mglib
_pkgname=mglib
pkgver=1.2.8
pkgrel=1
pkgdesc="Modules shared across all Papermerge project."
arch=('any')
url="https://github.com/papermerge/mglib"
license=('Apache')
depends=('python')
makedepends=('python-setuptools')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/papermerge/mglib/archive/v$pkgver.tar.gz")
sha512sums=('edfdc8d7ef861a055cfeba574e15d024053e8d6da2fc0e9696df0055bb91eb90be255b1570dd6d67697b9bfd1f205baf53e53d51a54c68354c44aed7eea2557f')

build() {
  cd "$_pkgname-$pkgver" || exit
  python setup.py build
}

# TODO: Fails with ModuleNotFoundError: No module named 'mglib'
# check(){
#     cd "$_pkgname-$pkgver" || exit
#     python test/run.py
# }

package() {
  cd "$_pkgname-$pkgver" || exit

  export PYTHONHASHSEED=0
  python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}

@Lucki, thank you for your hard work!
I will leave this ticket open maybe someone volunteers to continue your effort.

I might continue this but don't count on it just yet, since I have no experience MIT packaging yet and I also need t try papermerge first to see if I'm actually going to use it
Anyway, thank you for leaving thus here!

I'm coming from Paperwork and while I'm still happy with it I wanted to look into similar project with multi user capability.
So here's a list with things which aren't ideal IMHO:

In contrast to Paperwork or Docspell this uses a folder base approach and is really bad at it:

It's missing drag and drop or a context menu which adds five extra steps to every process. Automation helps here but you can't automate all of them in advance.

While cutting and moving pages also has the same problem it's nice to have it and I don't see another good way to do this. You don't see this feature that often.

Modal popup input fields don't get automatic text focus.

Having a multi user setup is pointless if only the superuser inbox is usable.

Fetching emails has to be a user setting.
Maybe watch subdirs for users?

Lack of feedback everywhere

Nothing happens when uploading finished.
Reloading the page and looking in the document is missing selectable text. Has processing failed or is it still on it? Do I have to retry? How do I let it process the document again? (It's possible to get some of the information in the log section but only if it was somewhat successful)

Scrolling documents to the bottom is a pain. The site has 4 (!) vertical and 3 horizontal scroll bars which activate one after another until you're finally at the bottom.
No fuzzy search. OCR often falsely recognizes characters or words or both and having a fuzzy search is crucial. A bit less than for Paperwork or Docspell because Papermerge uses folders but still important. (Search you demo for "Gesamtbetrag", which it indexed as "Gesantbetrag")
The right side menu (meta) is ugly compared to the other elements.
Automation based on pages will probably bite me later by ripping documents apart. Haven't tried that feature yet though.
Minor thing: No dark theme.

Hi @Lucki,
thank you very much for your feedback. I really appreciate it, because I heavily rely on this sort of feedback to improve the application. I am sort of "blindly" in love Papermerge, which make me unable to see many of its problems. I will open a separate enhancement ticket to keep track of your remarks.
The only thing that I want comment is regarding this point:

In contrast to Paperwork or Docspell this uses a folder base approach

this is not true anymore (well, at least in master branch :) ). In couple of weeks I will release newer version which will introduce tag management (I realized just very recently how useful tags really are!).

The apparent contra for paperwork ist that is appears to be only a desktop and no web based application. In that manner, it is simply not comparable with papermerge.
Docspell though could actually inspire further ideas for papermerge I guess.

thanks @amo13, I will have a look at docspell project for inspiration.

thanks @amo13, I will have a look at docspell project for inspiration.

Awesome! I am still trying to figure out which software I want to go for, considering papermerge, docspell and teedy, which appear to be the web-based candidates that are open-source and self-hostable.
About docspell: I like a lot the idea of having automatic correspondent identification and automatic tagging according to the content of a document. These would be awesome features in papermerge too. But I think I'm not going to install docspell because of the reliance on apache solr for full text search. I don't feel like spinning up and running this colossus just for full text search.
About teedy: what draws me to teedy is the simple fact that the project is already 7 years old, which makes me hope that it will not get abandoned in a year or so. Unfortunately, the drawback of young and innovative projects is that a lot of them end up abandoned before having the chance to actually flourish...

Unfortunately, the drawback of young and innovative projects is that a lot of them end up abandoned before having the chance to actually flourish...

My plans with papermerge span over several decades. It takes time to build something great.

Though I don't know exactly how, the admin package seems to be better integrated now and does not need to be moved in place any more. Therefore, I guess, this issue can be closed.


Confetti Visibility Interval Buster

add page visibility hook, clearInterval when not visible

@jmichaliga Can you check in src/hooks/usePageVisibility? It's not in the commit, so the build is failing. Should work once that's done.


EDEV-104: Wagtail 6.1 upgrade

Ticket URL: EDEV-104
About these changes
Upgraded Wagtail to v6.1 (and a few other patch versions of other packages)
Changed PASSWORD_REQUIRED_TEMPLATE to WAGTAIL_PASSWORD_REQUIRED_TEMPLATE as required in upgrade considerations.
Before assigning to reviewer, please make sure you have

[x] Checked things thoroughly before handing over to reviewer
[x] Checked PR title starts with ticket number as per project conventions to help us keep track of changes
[x] Ensured that PR includes only commits relevant to the ticket
[x] Waited for all CI jobs to pass before requesting a review
[x] Added/updated tests and documentation where relevant

Merging PR guidance
Follow docs\developer-guide\contributing.md
Deployment guidance
Follow docs\infra\environments.md

Hi @TNA-Allan , the places we use chooserUrls are in some JS overrides - I checked these and they're fine 😀


how to use self-host agent for azure pipeline

Hi, I want create a self-host agent and use it in pipeline. however, I had some problems importing the image.
the procedures are list in follow.

create a self-host agent according to the link : [https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops]

import the agent image, I do not know the right format
In other words, how could I replace lines 9 and 10 of the demo code with the self-host agent？

Thanks

Hi @tjinjin95
You should specify an agent pool name:
pool:
  name: 'AgentPoolName'

If you'd like to specify an exact agent in the pool you should do the following:
pool:
  name: 'AgentPoolName'
  demands:
  - agent.name -equals 'azurePipelineWindows'

Is this information helpful for you?

@egor-bryzgalov I still cannot specify the agent. The error log is as follows.
code:

first test:

second test:

capabilities:

Problem solved, could refer to the link: https://github.com/microsoft/azure-pipelines-agent/issues/2996