fix failing search test

Current develop branch tests fail.
457 Passing Tests
1 Failing Tests
Result range(32,0) is different from expected range(15,0) [/develop/XVim/XVim/Test/XVimTester+Search.m:136]
Fixed it.

What is this test case? Do you know why it had been passed before this fix??

All test pass as of develop branch 42fbe79664dc5e3260afb6d8675e7a80ffae3073.
The test is added by @yixiang at e7a89f4ea29f00d98009cf88af971e67833088cf.
But it doesn't pass.
I don't know #730 and #732 detail.
I just fixed it.

I think the root problem is that the e7a89f4 implementation doesn't obey the cursor position of the original Vim behaviour. So the test case is right. Actually it should be position at 17 and not 15 after the command. But at least it shouldn't be 32. I'll reject this PR but I'll fix test case to specify 17 as result and  comment out it to remember it.
Any comment?

So the intended behaviour of this search and replace function is as follows:
if (numReplacements > 0) {
  // position cursor on first non-blank of LAST range line
else {
  // position cursor on first non-blank of FIRST range line
}

To remedy this, I added the following code to the bottom of -[XVimSearch substitute: from: to: inWindow:]:
NSTextView* srcView = [window sourceView];
NSUInteger newPos;
if (self.numReplacements > 0) {
    // move to first non blank of last line
    newPos = [srcView.textStorage xvim_firstNonblankInLineAtIndex:to allowEOL:YES];
} else {
    // move to first non blank of first line
    newPos = [srcView.textStorage xvim_firstNonblankInLineAtIndex:from allowEOL:YES];
}

[srcView xvim_moveCursor:newPos preserveColumn:NO];

The difficulty with implementing this is that by the time the position is reset, XVim no longer knows how many substitutions have occurred, and resets the position, undoing the move in the previous code snippet.


🆕 Software Suggestion | CTemplar

Basic Information
Name: CTemplar
Category: Email
URL: https://ctemplar.com
Description
A highly respectable email service that is hosted in Iceland and has a collection of features that respect privacy, security, and anonymity of users.
Resources
CTemplar comparison table vs Protonmail and Tutanota
https://blog.ctemplar.com/ctemplar-comparison-table/
CTemplar open source code of their webclient
https://github.com/CTemplar/webclient

I don't think it should be recommended:

For example, if CTemplar receives a DDOS attack that we are not able to handle, we will switch to using Cloudflare.

I don't think it should be recommended:

For example, if CTemplar receives a DDOS attack that we are not able to handle, we will switch to using Cloudflare.

I think they mean they would temporally use Cloudflare services to stop the attack, I'll send them an e-mail asking for this and other thing I want to ask them, still their service is excellent in almost every sense, I don't think this is reason enough to not recommend them.
Here's a comparison chart that they provide on their website (comparing themselves against Protonmail, Tutanota, Hushmail and Gmail) for more details, I think they totally deserve to be listed.

I don't think it should be recommended:

For example, if CTemplar receives a DDOS attack that we are not able to handle, we will switch to using Cloudflare.

I'm not sure if there should be big worries about this given the fact that this scenario wouldn't be actively happening 24/7 and besides, if one really wants to transmit emails along with the worry of interception then their password encryption of email functionality can be used.

I don't think it should be recommended:

For example, if CTemplar receives a DDOS attack that we are not able to handle, we will switch to using Cloudflare.

I'm not sure if there should be big worries about this given the fact that this scenario wouldn't be actively happening 24/7 and besides, if one really wants to transmit emails along with the worry of interception then their password encryption of email functionality can be used.

Also I doubt you will be able to use their service during a DDOS attack therefore it's not like Cloudflare can have some relevant information about you.

@Mikaela Could you please give specifics about the "research required" label. Is there a criteria being based against in order to judge whether or not this particular email provider passes the standards established by privacytoolsIO?

@MystesofEternity I think most of the important part which is under the PTio criteria is described in their comparative chart here(https://ctemplar.com/ctemplar-comparison-table/). And I'm pretty sure they pass the criteria.

The new (draft) criteria can be found here.
They don't have DANE, and they're not registered EFF's STARTTLS-Everywhere list. They also need to publish a plan to deprecate TLS 1.0 and 1.1 on mail.ctemplar.com; they still accept them and the mail server doesn't enforce cipher suite preferences.
Also, no public-facing leadership or ownership on the website. You need to go to the GitHub repo to see they're a group of Pakistani nationals that own a web dev agency.
Lastly, trust. How much does the community trust them, not just one person, but a consensus.

The new (draft) criteria can be found here.
They don't have DANE, and they're not registered EFF's STARTTLS-Everywhere list. They also need to publish a plan to deprecate TLS 1.0 and 1.1 on mail.ctemplar.com; they still accept them and the mail server doesn't enforce cipher suite preferences.

Fair enough, I think this can be worked out, I'll try to communicate with them to see if it would be too difficult to do this.

Also, no public-facing leadership or ownership on the website. You need to go to the GitHub repo to see they're a group of Pakistani nationals that own a web dev agency.

Ehh, what's the problem with them being Pakistan people? I haven't seen anything on their GH profiles that makes me think that they are nationalists.

Didn’t say Nationalist, I said National.
national - noun: a citizen of a particular country, typically entitled to hold that country's passport.
"a German national"

Could you please give specifics about the "research required" label.

I don't know, I hope https://github.com/privacytoolsIO/privacytools.io/issues/977 will enlighten me on it and I guess I will be commenting there.

The whole company just screams untrustworthy. Offshore company in Seychelles talking about Icelandic privacy because they rented (virtual private?) server from Orange Website, while they are actually Pakistanis with address in Islamabad. What's stoping Pakistani authorities from coercing the founders?
Source: https://aretesol.com/portfolio-2/

I am the owner of CTemplar and I am a white male who resides in the USA.  Yes, I hired a developer from Pakistan.  I have also hired developers from Africa, Ukraine, France and South America.  I pick developers based on the quality of their code, I do not select developers based on the color of their skin or their nationality
My company was formed in Seychelles and servers in Iceland, exactly like Orangewebsite.com and flokinet.is.  These are strong locations
I made the planning decision to switch to cloudflare during DDOS attacks because I felt people need access to their email at all times.  However, after all the booters were taken offline I have not noticed any serious DDOS attacks.  Please do not view this statement as a challenge to DDOS my site, I am just sharing this for informational purposes.  I would be happy to revise my company's policy to not switch to Cloudflare during heavy attacks.
->If you review our technical specs by clicking this link.  DANE, TLS-RPT and MTA-STS are implemented and should reflect if this report is refreshed.  https://www.hardenize.com/report/ctemplar.com/1581171498
->If you review the two links below you'll see we are at 79% and Protonmail is at 72%.  I am not sharing this to slander PM, they are an excellent firm in every regard.  I am giving this as a reference point.
https://internet.nl/mail/ctemplar.com/320238/
https://internet.nl/mail/protonmail.com/320240/
->We plan on publishing our f-droid android app on March 2nd.
Please contact me with questions or concerns

I recall then when Ctemplar started, it turned out to be a bit of a rough start. That was because its marketing material existed mostly of pieces which were trying to hit on protonmail, and how Ctemplar was better. Then those articles were quickly removed, care to expand a bit on that?

I am the owner of CTemplar and I am a white male who resides in the USA. Yes, I hired a developer from Pakistan. I have also hired developers from Africa, Ukraine, France and South America. I pick developers based on the quality of their code, I do not select developers based on the color of their skin or their nationality. I will continue to hire developers based on their skill, without thought of their skin color.
My company was formed in Seychelles and servers in Iceland, exactly like Orangewebsite.com and flokinet.is. These are strong locations.
I made the planning decision to switch to cloudflare during DDOS attacks because I felt people need access to their email at all times. However, after all the booters were taken offline I have not noticed any serious DDOS attacks. Please do not view this statement as a challenge to DDOS my site, I am just sharing this for informational purposes. I would be happy to revise my company's policy to not switch to Cloudflare during heavy attacks.
->If you review our technical specs by clicking this link. DANE, TLS-RPT and MTA-STS are implemented and should reflect if this report is refreshed. https://www.hardenize.com/report/ctemplar.com/1581171498
->If you review the two links below you'll see we are at 85% and Protonmail is at 72%. I am not sharing this to slander PM, they are an excellent firm in every regard. I am giving this as a reference point.
https://internet.nl/mail/ctemplar.com/320238/
https://internet.nl/mail/protonmail.com/320240/
->We plan on publishing our f-droid android app on March 2nd.
Please contact me with questions or concerns

I have sent you an e-mail with more questions and you (or someone from your staff) told me something along the lines of "we will answer you briefly", would  you care answering those questions?

I recall then when Ctemplar started, it turned out to be a bit of a rough start. That was because its marketing material existed mostly of pieces which were trying to hit on protonmail, and how Ctemplar was better. Then those articles were quickly removed, care to expand a bit on that?

A company is composed of people who perform different roles.  It is my job to find the best person to perform each type of task.  I have no experience with marketing so I hired someone to do it for me.  I gave that person authority to execute their ideas.  After implementing this persons idea’s I realized it was a mistake, I picked the wrong person for the job, so I removed this person and removed their implemented marketing strategy.  I had someone message PM an apology on Twitter, they accepted the apology via Twitter.  Since then we have acknowledged Protonmail’s contribution to the security and privacy ecosystem. You can read about it on our blog.  https://ctemplar.com/ctemplar-recognizes-openpgpjs/ .  Personally, I feel they offer a wonderful service and their run by very qualified and capable people.  We are not enemies.  We are all on the same team trying to fight against the assault to people's freedoms.  The past marketing was a mistake, we apologized, and we posted truths about them.  I cannot change the past, but I can change how things are in the present, and in the future.

I have sent you an e-mail with more questions and you (or someone from your staff) told me something along the lines of "we will answer you briefly", would you care answering those questions?

Question: Who owns the company/organization? What percentage does each owner hold? (December 31 of prior year and current date)
Response: I have owned 100% of this company since it’s creation.
Question: Have you changed how information is processed and shared in the last year?
Response: We have not changed how any information was processed or shared.
Question: Do you share data – even “fuzzed” or “anonymized” data – with any of the owners/shareholders or any other company or organization server?
Response: I am the only shareholder, only owner and control all the voting rights. I don’t retain or share any information whatsoever.  The idea of sharing others information, even fuzzed or anonymized, is repulsive to me.  That’s why I created this company.
Question: Which components of your service are not open source? Where can we find the code for the open-source components?
Response: Our front end code resides here - https://github.com/CTemplar/webclient.  The back end is not open source.  This is what most every other service does I think.  If both our front and back end were made available, then someone could create perfect clones of the service.  Our back end code has no access to e2ee emails, it is ‘blind and dumb’ to what happens in the users browsers.
Question:  Have you had any independent audits in the last three years? Please share the dates of those audits and audit reports.
Response: We have had two independent audits done at launch. I’ll have to look for those.
Question: If you require sign-up or account creation, do consumers have easy access to tools to delete their data? Can they delete everything on the servers or just the local cache?
Response: Yes. Users can delete everything instantly; no backups are kept of anything that’s deleted.
Question: Is there a way for consumers to view any information you have collected about them?
Response: We collect nothing and when a person signs out of their inbox, all records and data of their visit are purged from our system. They are shown a screen telling them all their usage data is deleted every time they exit their inbox.
Question: What is your business model? How do you fund operations and make money?
Response: CTemplar’s business model is offering paid accounts.  We will never accept any donations, grants or investment from any outside source.  I can prove this by making my companies shareholder corporate data available.  But I won't be posting that publicly and I will require a signed NDA.
Question: Do you offer a transparency report?
Response: Yes it’s here: https://ctemplar.com/transparency-report-ctemplar/
Question: How is data secured (in transit and at rest)?
Response: In transit we use the latest encryption methods with a reasonable balance between security and compatibility. For CT to CT we apply PGP encryption on top.
Question: Who has access to customer data?
Response: The CC payment processor has access to user's payment information. Necessary developers have access to customers encrypted data so they can do their job.
Question: What 3rd parties have access to customer data?
Response: Our payment processor has access to people’s payment information if they pay with a credit/debit card. No other 3rd parties have access to any customer data whatsoever.
Question: What processes do you have in place if there is unauthorized access to data?
Response: Users data is encrypted when sent/recorded on the server.  If there is unauthorized access they will only be able to obtain encrypted content. We will be supporting E2EE encrypted metadata within the next few months also.
Question: What customer data is collected, how often, and in what level of identification?
Response: When you visit our website, your browser sends us your user-agent and IP address. When you leave our site, no records are kept of your IP address with an association with your account. The IP’s might be stored anonymously for up to 7 seven days if it is required to defend against attacks.  If you visit using our Onion site, then your real IP address is not seen.
Question: Will changes to your Terms & Conditions and privacy policies be communicated to end users at least 30 days in advance of any changes?  How will these changes be communicated?
Response: We will announce changes to our PP and ToS by posting it on Reddit/Twitter/Facebook.  I am happy to do it in other ways also.
Question: Do you plan on making your software available on F-droid? If not, why?
Response: Yes, we will make it available on F-droid on March 2nd.  We will be officially launching both the iOS and Android apps at the same time.  Both the Android and iOS apps will be open source on github.
Question: Another user commented on the issue about this entry from your blog (https://ctemplar.com/ddos-cdns-sri/) where you state that you will end up using Cloudfare services if you can’t handle a DDoS attack. Will you be using your regular service after the attack finishes? Are you planning on implementing something to not depend on Cloudflare?
Response:  We decided to remove that policy today; we will handle all DDOS attacks by our self.  We will never use Cloudflare or any CDN’s in any situation.

Yes of course, thank you for your patience. Please review this and let me know if you have any other questions

Don't worry, I understand you have a lot of other things to attend. By the way, is it better if I fill an issue on GH or send you an e-mail about troubleshooting? I have had a weird problem with your service lately.

Question: Do you share data – even “fuzzed” or “anonymized” data – with any of the owners/shareholders or any other company or organization server?
Response: I am the only shareholder, only owner and control all the voting rights. I don’t retain or share any information whatsoever. The idea of sharing others information, even fuzzed or anonymized, is repulsive to me. That’s why I created this company.

You do use ajax.googleapis, fonts.googleapis and gstatic. Do they not collect any kind of information? Would you be willing to change their services for an open-source alternative (awesome fork or awesome fonts) or even better hosting all your icons and/or fonts on your own?

Question: Which components of your service are not open source? Where can we find the code for the open-source components?
Response: Our front end code resides here - https://github.com/CTemplar/webclient. The back end is not open source. This is what most every other service does I think.

Yeah, it's pretty much and anyway it can't be proven that what's hosted it's the same as what's on GitHub.

Question: Have you had any independent audits in the last three years? Please share the dates of those audits and audit reports.
Response: We have had two independent audits done at launch. I’ll have to look for those.

Please provide URLs to these when you have time, it would be great to see them!

Question: What is your business model? How do you fund operations and make money?
Response: CTemplar’s business model is offering paid accounts. We will never accept any donations, grants or investment from any outside source. I can prove this by making my companies shareholder corporate data available. But I won't be posting that publicly and I will require a signed NDA.

I don't think this is necessary, and if anything, the staff are the ones who may be interested in seeing it, but I don't know.

Question: Who has access to customer data?
Response: The CC payment processor has access to user's payment information. Necessary developers have access to customers encrypted data so they can do their job.

So you do share some data with 3rd parties? Please don't take it bad, I'm just trying to provide some honest feedback, I really love your service!

Question: What customer data is collected, how often, and in what level of identification?
The IP’s might be stored anonymously for up to 7 seven days if it is required to defend against attacks.

Here you will find the criteria to list e-mail providers in case you want to check it with your team:
https://deploy-preview-1672--privacytools-io.netlify.com/providers/email/

->If you review our technical specs by clicking this link. DANE, TLS-RPT and MTA-STS are implemented and should reflect if this report is refreshed. https://www.hardenize.com/report/ctemplar.com/1581171498

I've refreshed it now as I have an account with Hardenize.

@Godfry what's the likelihood of Subresource Integrity on that google fonts usage?

@Godfry what's the likelihood of Subresource Integrity on that google fonts usage?

Also changing them for a FLOSS alternative or hosting everything on your website would be awesome.

@dngray We will be getting rid of the google fonts expeditiously.
@5a384507-18ce-417c-bb55-d4dfcc8883fe

You do use ajax.googleapis, fonts.googleapis and gstatic. Do they not collect any kind of information? Would you be willing to change their services for an open-source alternative (awesome fork or awesome fonts) or even better hosting all your icons and/or fonts on your own?

Those will all be removed immediately.  Thanks for bringing that to my attention.

So you do share some data with 3rd parties?

We share anonymous order numbers when needed to process refunds.  We share no other information with any 3rd parties.  We allow BTC and XMR also.

What does it mean that the IPs are stored anonymously? How do you anonymized them? Will you log the IPs of all your customers under such circumstances or just the ones of suspected attackers?

Never at any time do we link any account to any IP address.  There isn't a possibility for correlation. The IPs are stored in our logs for the minimum period of time we need to provide a stable service. These logs help to find harmful attack patterns and temporarily block the service. There is an automatic pruning and without manual intervention under normal circumstances.
If we are attacked by someone who also uses our service we would have no way to know.  And to be honest I don't care.  My only goal is to maintain steady service.
We are waiting to be registered under EFF's STARTTLS-Everywhere list,  And we have depreciated TLS 1.0 and 1.1.
Thank you for your comments and questions.  Kind regards to you,

Hi,
Thanks for your in depth reply @Godfry. I am glad to see a provider striving to meet as many good practices as possible.
There were a few things that irked me though (marketing related):

We use 4096 bit encryption. This protects users from current cryptography cracking techniques and future hypothetical attacks.

This implies RSA, public/private keypairs. We know that quantum computing at some point in the future make public/private key cryptography not built to be post quantum proof crackable.
Current literature on that topic suggests cracking this kind of encryption could be as early as 5 years away, but more conservative guesses put this at 10-15 years away.
The "and future hypothetical attacks" is not something you can realistically offer with your service.
On the same topic:
Do you plan to support AutoCrypt? I think this kind of out of band key sharing can be useful.
Ed25519 keys are becoming much more popular these days for PGP due to their size. It's also required as part of the spec.
WKD is also another really good way to do key discovery.

100% Anonymous

This is another assurance you can't really offer either. 100% means certainty without failure. The customer may very well compromise their own anonymity. Yes I know that wouldn't be your fault, but I would steer clear of making statements of certainty like that.

We never track your IP address, keep logs on your usage or record any identifying information at any time. We prove this by making our code readily available

I think what you mean to say is that you don't personally collect the data and therefore cannot reveal it/be made to reveal it. I think it would be better to say that you allow anonymous usage.

Encrypted subject

I found it rather strange that "encrypted subject" would be a paid feature. I assume as you use PGP, this would be compatible with Engimail or neomutt. What I have noticed in this there's a new standard being developed Protected Headers for Cryptographic E-mail.

4.2.  Confidential Subject
When a message is encrypted, the Subject should be obscured by
replacing the Exposed Subject with three periods: "..."
This value ("...") was chosen because it is believed to be language
agnostic and avoids communicating any potentially misleading
information to the recipient (see Section 7.1 for a more detailed
discussion).

Currently there's no standard what should be in the subject, for example Thunderbird by default uses "Encrypted Message" while neomutt defaults to "Encrypted subject" (can be changed with :set crypt_protected_headers_subject).
Are you using this method or some other method?
I do feel this should be a feature available to all. This would benefit your paid customers too.
Eg. Alice might join your service and become a paying customer. She might tell Bob about it and he joins too. However, Bob has decided to not to become a paying customer.
If Bob sends an email to Alice without encrypted subject (because it's not a feature available on free accounts), that actually isn't good for Alice, who is a paying customer.

@Godfry Greetings! First of all, thank you for offering such a wonderful email service and alternative to Protonmail.
Today, I just realized that CTemplar's 2FA security feature gives no backup codes and that worries me... alot. Thanks to have lost a couple of important accounts to a 2FA disaster before and me not being vigilant enough to keep backup codes.
Because of that, I've strictly kept backup codes for every single 2FA activated account that I have to this day.
Are there any plans to improve this feature so that it provides backup codes just like how other services do?

@MystesofEternity I think this is probably either a bug or maybe the devs implemented wrong something.  I think it is more appropriate to fill in an issue on their GitHub repository rather than asking the owner who is not in charge of programming.

@MystesofEternity please contact them directly for support issues, we want to keep this one about tracking related to privacytools.io

@dngray oh right, thanks for the heads' up about that.

@dngray

There were a few things that irked me

I apologize if my website's marketing upset you.  I can't change the past but I can change the wording going forward.

The "and future hypothetical attacks" is not something you can realistically offer with your service.

This was in reference to supercomputers' current ability to break RSA encryption.  4096 bit encryption gives greater future protection against supercomputer attacks in the future. If you still feel the wording should be changed I would be happy to change it.

We never track your IP address, keep logs on your usage or record any identifying information at any time. We prove this by making our code readily available.  I think what you mean to say is that you don't personally collect the data and therefore cannot reveal it/be made to reveal it. I think it would be better to say that you allow anonymous usage.

Point well taken, I’ll update our wording to reflect your comments.  Thank you.

I assume as you use PGP, this would be compatible with Engimail or neomutt. What I have noticed in this there's a new standard being developed Protected Headers for Cryptographic E-mail.

Yes we use PGP.  I’m not sure if we’re currently compatible with Engimail or neomutt but I’ll start looking into it.  Thank you for showing me the development in protected headers, I’ll follow that closely and implement it if it makes sense.

Do you plan to support AutoCrypt? I think this kind of out of band key sharing can be useful.

Yes we do plan to support AutoCrypt but I don’t have any idea when we’ll get to it.

100% Anonymous. This is another assurance you can't really offer either. 100% means certainty without failure. The customer may very well compromise their own anonymity. Yes I know that wouldn't be your fault, but I would steer clear of making statements of certainty like that.

Thanks for this suggestion, I agree with your comment and we’ll update our wording.

Currently there's no standard what should be in the subject, for example Thunderbird by default uses "Encrypted Message" while neomutt defaults to "Encrypted subject" (can be changed with :set crypt_protected_headers_subject). This may reveal information about the language used by the author, and what email software they use, hence why they chose .... Are you using this method or some other method?

We are currently showing *** for obfuscation because this is what is used for obfuscation for passwords as well.  We have no problem switching to using … if it seems the better decision.

Another question I had was do you support WKD/WKS.
We don't currently support WKD/WKS but it's on our development schedule.

I appreciate your comments and you'll see these changes reflected on the website when we do our next website update
Kind Regards,
Godfrey

@MystesofEternity
I've added your suggestion to our development schedule, it's a good suggestion.

@dngray
One more thing. If my website should be approved by privacytools.io then I request that it be at the bottom of the list of email services.
I think of the secure email services like soldiers in a military unit. Each competitive against each other to better each other but when the real enemy arrives they stand united each holding their shield to protect themselves and the person next to them.  For example some secure email services like PM or Tuta fight legal and cultural battles that my service benefits from. To continue the example if soldier is taken down (Like lavabit), there are others who can maintain the defense.  This analogy helps convey why I believe this industry benefits from several capable services who compete but stand together.  But I also want to show honor and respect for the other email services and being put at the bottom of the list helps to convey that.  Of course this is all contingent on being approved.
Kind Regards,
Godfrey

There were a few things that irked me

I apologize if my website's marketing upset you. I can't change the past but I can change the wording going forward. I'm sure many people share your same thoughts so this is an opportunity to improve things for everyone.

No problem. Sometimes missfires happen that we can reflect on later.

The "and future hypothetical attacks" is not something you can realistically offer with your service.

This was in reference to supercomputers' current ability to break RSA encryption. 4096 bit encryption gives greater future protection against supercomputer attacks in the future. If you still feel the wording should be changed I would be happy to change it.

I figured as much that was referring to a classical computing context. Personally I'd just omit the "and future hypothetical attacks". It's impossible to really know what the future will be and whether or not such current cryptography will be entirely useful if some major breakthrough occurs.
The statement in itself infers certainty that you know that it will be sufficient in all cases. I'd probably word it like this sidestepping the whole thing about current vs future.
"We use 4096 bit RSA encryption. This ensures the confidentiality of your messages."

I assume as you use PGP, this would be compatible with Engimail or neomutt. What I have noticed in this there's a new standard being developed Protected Headers for Cryptographic E-mail.

Yes we use PGP. I’m not sure if we’re currently compatible with Engimail or neomutt but I’ll start looking into it. Thank you for showing me the development in protected headers, I’ll follow that closely and implement it if it makes sense.

The method that Enigmail/Neomutt uses seems to be the same. It seems it was not part of the AutoCrypt spec, however the RFC Protected Headers for Cryptographic E-mail does seem to be drafted by AutoCrypt group.

Do you plan to support AutoCrypt? I think this kind of out of band key sharing can be useful.

Yes we do plan to support AutoCrypt but I don’t have any idea when we’ll get to it.

I did notice this:

Tutanota also has it on their roadmap (Tutanota)
from /u/ProtonMail on /r/ProtonMail they do mention "has significant security weaknesses and therefore we don’t feel it’s ready for implementation", but don't elaborate.

My guess is it would have something to do with the warning that Posteo gave on the bottom of their announcement: "Security recommendations for implementing Autocrypt".

New: Easy email encryption with Autocrypt and OpenPGP header (Posteo)

Currently there's no standard what should be in the subject, for example Thunderbird by default uses "Encrypted Message" while neomutt defaults to "Encrypted subject" (can be changed with :set crypt_protected_headers_subject). This may reveal information about the language used by the author, and what email software they use, hence why they chose .... Are you using this method or some other method?

We are currently showing *** for obfuscation because this is what is used for obfuscation for passwords as well. We have no problem switching to using ... if it seems the better decision.

Yes I'd use ... as that's what is defined in the specification, though there is more to it than just that. It would also be good to see CTemplar listed on the implementation status page, if/when you get it complete.

Another question I had was do you support WKD/WKS.
We don't currently support WKD/WKS but it's on our development schedule.

Excellent!

I appreciate your comments and you'll see these changes reflected on the website when we do our next website update

Awesome.

@dngray
One more thing. If my website should be approved by privacytools.io then I request that it be at the bottom of the list of email services.

The initial ordering was based on coverage of the criteria. We had then decided to just add new ones as they come along. That being said, there had been some suggestion of using alphabetical ordering to be more fair.

Of course this is all contingent on being approved.

After the few marketing related things you mentioned above I don't see anything standing in the way.

I was looking at CTemplar and it really seems to be an alternative but let me just get off the subject of this discussion to talk about the free plan.
Only three e-mails per hour? Really? Not even for a simple test before you migrate from another server is that enough.

I was looking at CTemplar and it really seems to be an alternative but let me just get off the subject of this discussion to talk about the free plan.
Only three e-mails per hour? Really? Not even for a simple test before you migrate from another server is that enough.

This is not the place to discuss this kind of things, the issue should be about if we should or not list the provider based on other things, not on how much the free plan can do.  Providers from Silicon Valley make you think you don't have to pay anything but in fact they sell your personal data to keep the service running, almost every e-mail provider out there, that respects your privacy is paid, ProtonMail is an exception since they are the most known one and can afford to do it, CTemplar is a really new project, be considerate.

I am the owner of CTemplar and I am a white male who resides in the USA.

Question: Who owns the company/organization? What percentage does each owner hold? (December 31 of prior year and current date)

Response: I have owned 100% of this company since it’s creation.

? "...a white male who resides in the USA" only narrows the ownership down to @ 50% of US residents. "I" could be almost anyone. The question is seeking verifiable owner and company information.

This is true actually and we do have a requirement for all providers:

"Public-facing leadership or ownership."

We do not allow for random .onion e-mail services (often suggested) because nobody really knows who owns them ie. could be your worst adversary.
It does say on their transparency page:

We will provide our Seychelles company incorporation documents and ownership information to media outlets, to demonstrate that we are not owned by any government or corporation. This will be issued under a non-disclosure agreement. Please email<EMAIL_ADDRESS>for more information.

I would have like to have been able to verify this from a public source.

@fabianski7 Yes our policy is 10 emails per hour to 3 different non-ctemplar recipients. Unlimited CTemplar to CTemplar.  We might make this unlimited to any other encrypted email service in the future.  The reason for this policy is because people have created hundreds of free accounts and would use them to email countless sextortion emails.  Limiting outgoing emails is the best way to reduce that kind of abuse.   I believe Protonmail arrived at that same conclusion and have a 10 email per hour limit also.  If you have any suggestions about how to prevent abuse while offering a free anonymous service, please let me know.
@LizMcIntyre After it was falsely circulated that I am a Muslim, people flooded our support inbox with insults against Muhammad and statements about how only white people are capable of writing code.  I hope that explanation explains why I responded like I did.  Our transparency policy states that we will verify ownership and company information to organizations that sign NDA's.
Why dont I want my name/picture all over the website?  My developers wrote the code. We used the PGP code Protonmail.com maintains.  We were inspired by countless security professionals who never receive any recognition at all.  I do not feel I deserve any special honor, glory or reorganization.  Someday If I personally make exceptional contributions to the security and privacy industry I might feel I earned putting my name & face all over my website.  Then maybe everyone would see that I could have made a fortune as a male model?  Maybe everyone would want to have cutting boards made with a picture of my six pack abs on it because my ab's are so rock solid?  I am just making a joke.  I understand your concern and I will distribute my identity as it is needed.
@dngray  I would be more than happy to send yourself or any Privacytools.io admin my incorporation documents along with my passport, drivers license, and any other documents that are desired.  I only ask that an NDA is signed beforehand.  You can contact myself or the legal team, they have fulfilled several of these requests so far.
I'll be done with the comments you outlined shortly.  I am also going through the rest of my website and trying to change/revise anything else that could be viewed negatively.  If feel you would be rightly insulted if I made the changes you stated, but did not change similar writing elsewhere.  I am just being careful to make sure I treat your time respectfully.  I'll be done shortly.
Kind Regards to you all,

@Godfry
You have an insecure connection on the registration page.
See: https://i.imgur.com/US88FCI.png
http://api.ctemplar.com/clock-image/6348e926b78d82958d6e2ee5ac3c10e712493c22/
https://developer.mozilla.org/en-US/docs/Web/Security/Mixed_content
If security.mixed_content.block_active_content is active (which is the default) it won't be possible to see the clock image.

It seems that using user.js from ghacks breaks most of the site.
Accessing https://mail.ctemplar.com/mail/settings/dashboard-and-plans doesn't seem to be an easy task. See the amount of errors in the console: https://i.imgur.com/5bl3kLk.png
They repeat themselves everywhere. Some pages don't even open.

@fabianski7 I have tested using user.js from ghacks and it worked perfectly, there is already a fix which fixed most of the icons issue using that profile.
If you still face the issue, please feel free to report an issue here https://github.com/CTemplar/webclient/issues
Thank you.

@fabianski7 about the registration page insecure, we are going to remove captcha completely, the fix is already on its way and will be availble soon.
If you still have any issue, please report here :
https://github.com/CTemplar/webclient/issues

@fabianski7 Your comments are noted and they will be corrected.
I want to discuss something with you Fabiansky7.  You were asked by a moderator not to post these sorts of posts to this channel.

This is not the place to discuss this kind of things, the issue should be about if we should or not list the provider

I feel that an unbiased reader of this chain of comments would state that your attitude is aggressive and your goal is to belittle my service and cause distrust.  I feel this seems especially true after taking into account your disobedience of the moderator, and disrespect for the Privacytools.io website when you use it as a platform to report/solve bugs.
If your intent is to belittle and discredit my service, I will give you a public platform to do it.  I will publish it on my website in the "Zero-Censorship" area.  My service has made mistakes and has had bugs.  I will not hide from them.  Please email my support team if/when your ready to post your content to my website.

@fabianski7 registration page insecure issue has been fixed as we have removed captcha for free signup. https://github.com/CTemplar/webclient/releases/tag/v2.1.3

If your intent is to belittle and discredit my service, I will give you a public platform to do it. I will publish it on my website in the "Zero-Censorship" area.

Well, that was not my intention. I enjoyed the space here because since you are the owner of the service, it would be easier to report bugs.
I'm sorry if I left that impression.

[...] You were asked by a moderator not to post these sorts of posts to this channel.

I didn't see any admin complaining about what I posted (though this really isn't the place for it).
My intention was only to help with both sites.

@fabianski7  You responded kindly, and you kept the door open for continued friendly communication. That is a rare approach and I appreciate it.  Sometimes in online communication it is difficult to gauge someone's mood and disposition because you can not see their expressions and hear the intonation of their voice.  In this instance I feel I misunderstood you, I apologize.  And you are correct I have often said that anyone is free to report bugs directly to me.  Ultimately nothing matters except protecting people against privacy abuse. I think that's something you understand, thank you for your comments.
Your username reminds me of the football player named Fabianski, and that reminds me of a joke.  Long ago I played football. During one game I wore a white uniform.  Before the game I ate a chocolate bar to increase my energy.  I ran onto the field and realized that I had mistakenly sat on the chocolate bar, I had a big brown stain on the back of my white shorts.  I know that seems unlikely but before games it becomes very chaotic.  At the time it was embarrassing but I think it is funny now.
FYI - I'm almost done combing through my site removing rough edges.

@dngray I have completed all the wording changes to the website that I feel you indicated as well as other changes I feel needed to be made.  If you notice other things, please give me the opportunity to correct them and I mean no disrespect to you or the Privacytools team.  We have also started working on implementing autocrypt and we should be done with that within the next few weeks.
Respectfully, I submit my email service to be reviewed so that it can be listed on the email review page

@dngray You may want to look into this https://www.reddit.com/r/ctemplar/comments/flh2hg/website_hosted_on_same_server_as_mail_server/

@zack-95   If you read through the above comments you will see that using Privacytools.io as a platform to report bugs and concerns with our service is not acceptable.  This policy has nothing to do with my service, it has to do with protecting the reputation of Privacytools.io and making sure it is never viewed as a support website for other services.  If Privacytools.io is viewed as a support platform for services like mine, it reduces their ability to help people protect their privacy across all their needs.
We are preparing a response to your comment on Reddit and will post it shortly.  If you have other concerns or questions please direct them to myself or my support team.
If your looking for a platform to make negative comments about my service I would be happy to publish your unedited review to our zero censorship page.  In this way, you will benefit from greater visibility.

It is totally related, as it relates to a security risk, and I'm sure PTIO would not want to list a provider with such poor privacy practices that put their customers at risk.

@Godfry This isn't about reporting bugs. It's about the technical competence of the company in general. I am going to be candid here.
If you run nmap ctemplar.com, you get something like this:

This is of high concern, especially considering that you are a secure email service provider.
Furthermore, if we look for DNS records on your domain, all the domains and subdomains lead to one IP, suggesting there is only one server: the one in the screenshot above.

Your WordPress site (the CTemplar site itself), Jenkins and mail is all hosted on this one server, and WordPress has not had the best history of vulnerabilities to say the least (Kevin Beaumont, Senior Threat Intelligence Analyst for Microsoft Threat Protection went so far as to call it a "really good Emotet distribution tool with an okay content management system bolted on"). If we go to port 3000, we see an exposed Nextcloud instance, SSH is open on port 22 in password auth mode. This is obviously unacceptable for any sort of service, but especially one that claims to offer "armoured email".
I don't know how much you yourself understand about this, but you ought to get some people onboard that have professional knowledge around cyber security. Running a secure email is not a joke and bad cybersecurity is not just a "light inconvenience".  These comments and posts are very well-placed as PTIO (or really anyone) should not be recommending or using insecure providers for the sake of themselves.

Very wise words @zack-95 and @VigilantSwanson.  @dngray It's worth reading and then re-reading all of that to be sure you get it all.
@Godfry Privacytools is only relevant because of the community. If the community wants them to be a support platform that's what they will be. So STFU about that.
Along with the wise comments listed above here are more concerns:

The state department issued a level 3 travel advisory because Pakistan has terrorists, kidnappers and insurgent groups.
There are 60,000 Mexicans incarcerated in US jails for rape, murder and drug dealing.  How do we know your Mexican employee is not involved in that?
  How do we know your LGBTQ non-binary Ze/Hir wont use the service to abuse children?  A responsible CEO would at least help it get treatment for it's illness.

Privacytools will never list a service run by terrorists, rapists and gender confused child abusers.  The community rejects your proposal.

@LordNikon2x Your blatantly racist reply is completely unacceptable. You are profiling people based on what other members of their ethnic group are doing. These developers likely commited no crime and yet are being blamed, just because they were born this way. This is a place for civil discussion, not for ad hominems and racism. I may disagree and have objections with the security practices of ctemplar but to be so rude and discriminatory is completely uncalled for and unessesary.
Please have some decency and debate with facts, not misleading blanket statistics. Furthermore, speak for yourself. You have no right as a racist to speak for any members of this community and clearly do not understand or have knowlege of the community.

I'm so impressed with what was said by @zack-95, @VigilantSwanson and @LordNikon2x.
I am a student at a highly esteemed university.  I researched everything they said and agreed to it all because it is the truth.
The email page of PTio should stay the way it is and no new sites should be added.The community and PTio came together and did a great job! Together we can achieve dreams!

@dngray I have completed all the wording changes to the website that I feel you indicated as well as other changes I feel needed to be made.

I shall give it a read in the coming days.

If you notice other things, please give me the opportunity to correct them and I mean no disrespect to you or the Privacytools team.

Sure.

We have also started working on implementing autocrypt and we should be done with that within the next few weeks.

Cool.

@dngray You may want to look into this https://www.reddit.com/r/ctemplar/comments/flh2hg/website_hosted_on_same_server_as_mail_server/

Thanks for posting the link.

@zack-95 If you read through the above comments you will see that using Privacytools.io as a platform to report bugs and concerns with our service is not acceptable.

While I would agree with this sentiment it is concerning to see so many services open to the internet, particularly things like Jenkins which have quite a history in the past.
Another thing worth noting that meeting the criteria does not mean a certainty in being listed. It is a minimum baseline and a guide to what we look for.

We are preparing a response to your comment on Reddit and will post it shortly. If you have other concerns or questions please direct them to myself or my support team.

I shall be looking forward to reading that.

It is totally related, as it relates to a security risk, and I'm sure PTIO would not want to list a provider with such poor server infrastructure security practices that put their customers at risk.

Spot on.
@LordNikon2x
Please don't bother posting anything like that again.
@Jeremy-Stanford
This isn't the place for that.

Nice "Zero Censorship Policy"...
https://ctemplar.com/zero-censorship-policy/

After some discussion we've decided not to add CTemplar at this time.
The reason being we do not like to provide information which cannot be verified by public sources. We don't allow anonymous companies to provide services because it involves people trusting an unknown entity with their data that cannot be verified. If the company fails or does something disastrous there is no recourse.
We won't be signing any NDAs regarding this, as it would mean we cannot reveal what we learn, and thus puts it on the community to trust us instead of the company they're doing business with.
I do however want to thank @Godfry and his developers for making the improvements we suggested. I also want to thank those who contributed meaningful replies.

@dngray

Another thing worth noting that meeting the criteria does not mean a certainty in being listed. It is a minimum baseline and a guide to what we look for.

If it's Privacytools.io decision not to list my service I understand.  However, I would like to be able to say that I meet your criteria.  I will provide all the company verification documents to the privacytools.io team without a signed NDA.   If I provide those documents to your team, can I say that I meet your criteria?   You don't have to list my service

@VigilantSwanson
1.- It's true we have a single public facing IP, all our services behind are properly virtualized and totally isolated from each other. This applies to all public and private services, including those you've discovered with nmap and WordPress.
Even if we didn't consider enabling remote access to different internall tools using a port other than 80/443 a security issue at all, we have closed them to minimize discoverability of the different tools we use.
2.- We know SSH is listening in its standard port and its settings. We have decided to not change it because we consider our implementation protects our systems from brute-force attacks and any other unauthorized access we can imagine.
Kind Regards,

However, I would like to be able to say that I meet your criteria.

With this you'd be saying: "we meet the criteria but we don't meet the criteria".
This creates problems as other providers would seek the exemptions to say they meet the criteria when they in fact don't. This would in turn dilute our purpose and compromise our mission. Our endorsement and branding would become meaningless.
It is likely to confuse users as well. They're likely to open many issues with both you, and us about why they are not on the PrivacyTools site, when they apparently meet the criteria.

I will provide all the company verification documents to the privacytools.io team without a signed NDA. If I provide those documents to your team, can I say that I meet your criteria?

The issue is with that we would have to distribute them on our site. We would have to provide some kind of public verification or reference that what we say is actually true. This is what gives PrivacyTools it's authority over other sites who simply just say X is good without any kind of validation or peer review.
There are many sites which endorse many things without reason or reference. What gives PrivacyTools it's reputation is the fact that discussions about what is added happen transparently, in public such as on GitHub. People can track the discussion and reasoning and use it in future debates as to why/why not a specific product should be used.
If we make recommendations with "secret sources", it encourages people to accuse us of being biased, bribed, compromised etc. We then would get this pollution on blogs, social networking websites and in comments on our own forums of discussion. It would confuse people and overall they would trust us less.
Members of the community would be able to clearly see that there is information they are "not allowed" to know. All sorts of conspiracy theories would be speculated. Members of our community have typically had their trust abused previously by large companies seeking to make a profit off their private data, as well as governments claiming to be invading their privacy for their own safety.
The other thing to note is, we're all people with regular jobs (mostly in IT). PrivacyTools is certainly a community project that depends on our spare time, and public donations. As a result there was a significant discussion Preventing Privacytools conflicts of interest - ensuring Privacytools integrity, which resulted in us creating a Conflict of Interest Policy, this is to provide some recourse should a team member work at a company which is also a recommended product or wants to be a recommended product.
From an legal standpoint I would certainly not be distributing any kind of documents covered under an NDA normally for other parties. From an ethical point I would refuse to posses such documents unless I had authority to distribute.
If you did give such permission, then you'd be better off distributing them yourself.

@dngray I understand.  Thank you for explaining.
If you tell me how Soverin & Disroot met Privacytools.io criteria in this regard, I will do exactly the same.  I think that would be the simplest way to resolve this.

Could you please tell me the criteria? Once I know exactly what you're looking for I'll meet it.

Sure, the criteria is available on our site https://www.privacytools.io/providers/email/#criteria

Could you tell me how Soverin and Disroot met the criteria? I would like to use them as examples to be sure I provide a complete response.

What part specifically? Both of these are public. Both providers are listed on KVK Disroot and Soverin. More information about KVK. You cannot register in the KVK without your legal name and contact details.
Both Soverin and Disroot also have have a presence on social media, which means we get to know something about the people behind the service. Eg. @muppeth I've often seen around on Github.
Soverin have relevant information about them located: https://soverin.net/about
There is a higher trustworthiness associated with a company being run in the same location as where the employees reside.
They also do use their real names, when promoting their product, and likewise on Twitter: Ivo Fokke, Patrick, Andre Meij.

In addition, ctemplar does not support IMAP, SMTP or JMAP.

In addition, ctemplar does not support IMAP, SMTP or JMAP.

This is not a requirement. See Tutanota. It's a best-case option.

You're right. btw, I saw on reddit that POP3/IMAP/SMTP support will be added next month.
https://www.reddit.com/r/ctemplar/comments/fjtiou/new_features_development_schedule/

@dngray
I'm not trying to make you list my site.  I am only discussing this because I would like to meet your criteria.  Based on what you've shared with me I feel my company meets the criteria of having  "Public-facing leadership or ownership."

Soverin and Disroot also have have a presence on social media

My service does also. Facebook, linkedin , Twitter

Both providers are listed on KVK Disroot and Soverin.

My service is in the Dun & Dradstreet Global Database. I think that's at least equal to the KVK number. You can lookup and verify my company in the database if you use the number “56-137-7531” and search it by the country Seychelles.
I use an Alias because I've noticed many people in the security ecosystem use aliases.  if that's what's required then I would be happy to use my real name.

Pleased to meet you, Paul
While aliases and privacy are respected, when dealing with an actual company that provides those privacy services, as @dngray said, I'd like to know who I'm dealing with. Hiding who is behind it actually makes me trust it less.

@zack-95
I see your view. I'm happy to provide any verification that you all want. I've added my name to github, and picture & name to http://keybase.io/, I've verified myself with keybase's encryption.
From my view (a view gained from comments on this thread/reddit & direct email) I felt this community was trying to find a reason to disqualify my service because of hate toward Mexicans, Pakistanis and those of the LGBTQ community. Thank you for reopening the issue. Please give me a chance to respond to concerns before denying & closing the thread.
If the issue really is the nationality and gender of my employees then let's have out with it.  As some of you who have emailed me will know, I will not give my opinion and will respond with academic studied that I think was conducted well.  I would rather discuss this openly instead of having it be talked about in secret.
Kind Regards,

From my view (a view gained from comments on this thread/reddit & direct email) I felt this community was trying to find a reason to disqualify my service because of hate toward Mexicans, Pakistanis and those of the LGBTQ community.

This is certainly not the view by the PrivacyTools team. We would never disqualify a provider based on these things. We do in fact have a Code of Conduct related to this.

If the issue really is the nationality and gender of my employees then let's have out with it.

Certainly not, and as such I have not mentioned it, because it is not something we use in our deciding factors.

@dngray Thanks for the response:)  I'll take your advice and ignore the trolls.
Let me know anything else you need.

@dngray @Godfry Any update on this? It has been a while, and the issue is open.
CTemplar has made some progress, so ig it would be worth looking to add them to https://privacytools.io/

have owned 100% of this company since it’s creation.

I am the only shareholder, only owner and control all the voting rights. I don’t retain or share any information

CTemplar’s business model is offering paid accounts. We will never accept any donations, grants or investment from any outside source. I can prove this by making my companies shareholder corporate data available. But I won't be posting that publicly and I will require a signed NDA

https://ctemplar.com/help/answer/do-you-offer-imap-2/
IMAP may be arguably not required for adding to the list, but the fact that it was unequivocally promised soon at least twice over the last year gives a hint that the company might not have sufficient resources or may have difficulties of unknown nature, rendering its future questionable.

This discussion has been going on for a long time.
How about we end this discussion?
CTemplar does not implement CSP.
I am against adding it.
https://observatory.mozilla.org/analyze/mail.ctemplar.com

Doesn't look good 😥🥺 https://cyber-privacy.net/ctemplar-catastrophic-incident-with-complete-data-loss-july-2021/

"We cannot restore data from backups because we do not keep backups for security reasons" now that one is new

Doesn't look good 😥🥺 https://cyber-privacy.net/ctemplar-catastrophic-incident-with-complete-data-loss-july-2021/
Someone forwarded me this, but I can't attest for its trustworthiness. Can someone confirm this ?


Issue [Mac]: Opening As Login Item Causes Blank Display

If I add this app to my login items, when it opens up it gets stuck on the blank white screen which normally occurs before it goes to the blue splash screen

I see the same thing on Linux using i3wm. If anybody has any insight on this I'd appreciate it, haven't spent much time investigating. My workaround is that I launch the app from a bash script that waits a few seconds before running it. So far the few extra seconds has prevented the problem completely.


problemCondition not recognized if opponent has the last move in the "correct" variation

In this "copy this joseki"-style problem, Black has the first move, but White ends the sequence. Although the last node has a GB[] property, the STANDARD_PROBLEM viewer does not show the check mark. If I remove the last move and add GB[] to the B[dn] move, it works.
(;AB[mc][pd][qd][oe][of][pf][cp]AW[lc][oc][md][od][ne][pe][eq]C[Copy this joseki.]
CA[UTF-8]FF[4]GM[1]PL[B]SZ[19];B[gq];W[do];B[dp];W[ep];B[eo];W[fo];B[en]
;W[gp];B[dn];GB[]W[hq])

Normally the player whose turn it is to solve the problem will also have the last move, but there are problems such as this where I'd like to show a sequence for both, so the opponent can have the last move.

What if there was a problem-playthrough option?
I think that it's still correct to put GB on the Black B[bn] move -- since that represents the correct sequence for black -- the player playing the problem. However, I supposed could add an option that would play the last opponent's move -- if such a move existed.
It's possible that I could make this the default behavior too.

I agree GB[] should be on B[] nodes, but the same applies with problemConditions: { C: ['RIGHT'] }.
The way EasyGo for iOS and Guo Juan's training system implement this - as far as I can see -, and I think it makes sense,is that if the final node in a variation is marked as correct, then the problem is considered correct, no matter which other properties that node has.

How do EasyGo and the GuoJuan training system deal with wrong-variations?  Usually, problems list not only the correct variations but also some incorrect variations with some explanation.

 If you click on a position that's not in the tree, it just briefly flashes a red cross at that position and doesn't play the stone.
See the first demo video on http://www.easygo361.com/videos/
I also like, as shown in that video, how in the problem exploration moves that lead to a correct node are shown with green dots, whereas moves that don't are shown with a red dot.


Disable diff when executing idempotent check

When performing the idempotence check, the regexp fails when output
contains --diff output.  Run the reconverge without --diff.
Fixes: #554

👍


feat: optimize CoalesceBatches in limit

Which issue does this PR close?
Closes #11980.
Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

The issue explained in https://github.com/apache/datafusion/issues/9792 was resolved with the implementation of https://github.com/apache/datafusion/pull/11652. This fix handles the problem related to waiting for the coalescer buffer to fill when a Limit -> ... -> CoalesceBatches pattern exists. The approach was to push down the limit (fetch + skip) into CoalesceBatches and eliminate the limit when it was no longer needed.
With https://github.com/apache/datafusion/pull/12003, it appears that additional corner cases are being addressed. It further refines the process by pushing limits as far down the execution plan as possible and removing any redundant limits.
It seems that these recent improvements already address the objective you're aiming for, without the need to define a constant thresholds. I think there is no difference between using a limit without coalescing and using a coalesce that can internally handle limits.
I am curious about your thoughts. Do you still see a need for additional optimization? If so, could you provide an example scenario or a test case that would help us discuss this further?

The issue explained in #9792 was resolved with the implementation of #11652. This fix handles the problem related to waiting for the coalescer buffer to fill when a Limit -> ... -> CoalesceBatches pattern exists. The approach was to push down the limit (fetch + skip) into CoalesceBatches and eliminate the limit when it was no longer needed.
With #12003, it appears that additional corner cases are being addressed. It further refines the process by pushing limits as far down the execution plan as possible and removing any redundant limits.
It seems that these recent improvements already address the objective you're aiming for, without the need to define a constant thresholds. I think there is no difference between using a limit without coalescing and using a coalesce that can internally handle limits.
I am curious about your thoughts. Do you still see a need for additional optimization? If so, could you provide an example scenario or a test case that would help us discuss this further?

Thanks for providing the background on this optimization. I looked into the issues you mentioned and it seems they've been resolved exactly as I hoped. Great job! I'll reference the information you compiled in my issue.


Add wayland-session and xsession files

It would be nice to have a wayland-session file in /usr/share/wayland-sessions/ and an xsession file in /usr/share/xsessions/, so that display managers can detect sway and display an option to start it.
Those files could look like this for example:
[Desktop Entry]
Name=sway
Comment=SirCmpwn's Wayland window manager
Exec=sway
Type=Application

(note: I haven't tried out if the file above actually works)

I agree, but only if we see it tested.

Nice. I'm gonna try it out with GDM.

I've tried it with the following files:
/usr/share/wayland-sessions/sway.desktop
[Desktop Entry]
Name=Sway
Comment=SirCmpwn's Wayland window manager
Exec=sway
Type=Application

/usr/share/xsessions/sway-xorg.desktop
[Desktop Entry]
Name=Sway (Xorg)
Comment=SirCmpwn's Wayland window manager
Exec=sway
Type=Application

Works fine for me when using GDM. Note that the xsession file has to have a different name then the wayland session file.

Cool. Pull requests welcome.


Basic backend (GET and POST recipes)

Currently there is no auth for POST recipes so no usernames
Note that the url is localhost:8000/api/{api endpoint} and use the params for the post request in postman

comment schema implemented with content and time stamp


Carry minor changes for microsoft.sql 2021 11 01 preview

MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Most of the diffs are not real changes, only changing the order: in our script, we started to sort the content in the auto-generated manifest file
We ignore MISSING_APIS_IN_DEFAULT_TAG
We ignore ❌ R4009 - RequiredReadOnlySystemData because SQL does not fully support SystemData property. This exception has been approved by ARM in 2021.
We remove the legacy "ReplicaitonLinks_unlink" API. It is a redundant and dummy API; no user uses it.

Changelog
Add a changelog entry for this PR by answering the following questions:

What's the purpose of the update?

[ ] new service onboarding
[x] new API version
[ ] update existing version for new feature
[ ] update existing version to fix swagger quality issue in s360
[ ] Other, please clarify

When are you targeting to deploy the new service/feature to public regions? Please provide the date or, if the date is not yet available, the month.
When do you expect to publish the swagger? Please provide date or, the the date is not yet available, the month.
If updating an existing version, please select the specific language SDKs and CLIs that must be refreshed after the swagger is published.

[ ] SDK of .NET (need service team to ensure code readiness)
[ ] SDK of Python
[ ] SDK of Java
[ ] SDK of Js
[ ] SDK of Go
[ ] PowerShell
[ ] CLI
[ ] Terraform
[ ] No refresh required for updates in this PR

Contribution checklist:

[x] I commit to follow the Breaking Change Policy of "no breaking changes"
[x] I have reviewed the documentation for the workflow.
[x] Validation tools were run on swagger spec(s) and errors have all been fixed in this PR. How to fix?

If any further question about AME onboarding or validation tools, please view the FAQ.
ARM API Review Checklist

Applicability: :warning:
If your changes encompass only the following scenarios, you should SKIP this section, as these scenarios do not require ARM review.

Change to data plane APIs
Adding new properties
All removals

Otherwise your PR may be subject to ARM review requirements. Complete the following:

[x] Check this box if any of the following apply to the PR so that label "WaitForARMFeedback" will be added automatically to begin ARM API Review. Failure to comply may result in delays to the manifest.

Adding a new service
Adding new API(s)
Adding a new API version
-[ ] To review changes efficiently, ensure you are using OpenAPIHub to initialize the PR for adding a new version. More details, refer to the wiki.

[x] Ensure you've reviewed following guidelines including ARM resource provider contract and REST guidelines. Estimated time (4 hours). This is required before you can request review from ARM API Review board.

[x] If you are blocked on ARM review and want to get the PR merged with urgency, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.

Breaking Change Review Checklist
If any of the following scenarios apply to the PR, request approval from the Breaking Change Review Board as defined in the Breaking Change Policy.

[x] Removing API(s) in a stable version
https://github.com/Azure/azure-rest-api-specs/pull/18005#issuecomment-1055926162
We remove the legacy "ReplicaitonLinks_unlink" API. It is a redundant and dummy API; no user uses it.

[ ] Removing properties in a stable version

[ ] Removing API version(s) in a stable version

[ ] Updating API in a stable or public preview version with Breaking Change Validation errors

[ ] Updating API(s) in public preview over 1 year (refer to Retirement of Previews)

Action: to initiate an evaluation of the breaking change, create a new intake using the template for breaking changes. Addition details on the process and office hours are on the Breaking change Wiki.
Please follow the link to find more details on PR review process.

SQL does not support systemData until now. The feature will be completed TBD.
❌ R4009 - RequiredReadOnlySystemData

This error is expected. We need to ignore it.
We remove the dummy ReplicationLinks_Unlnk API endpoint. This API did not in use for many years.

Rule
Message

❌ MISSING_APIS_IN_DEFAULT_TAG
The default tag does not contain all APIs in this RP. Please make sure the missing API swaggers are in the default tag.readme: specification/sql/resource-manager/readme.mdjson: Microsoft.Sql/stable/2014-04-01/replicationLinks.json

Rule	Message
❌ MISSING_APIS_IN_DEFAULT_TAG	The default tag does not contain all APIs in this RP. Please make sure the missing API swaggers are in the default tag.
readme: specification/sql/resource-manager/readme.md
json: Microsoft.Sql/stable/2014-04-01/replicationLinks.json

    "administratorLoginPassword": {

Can you annotate it with x-ms-secret?
https://github.com/Azure/autorest/blob/main/docs/extensions/readme.md#x-ms-secret

Refers to: specification/sql/resource-manager/Microsoft.Sql/preview/2021-11-01-preview/ManagedInstances.json:854 in b3060af. [](commit_id = b3060afadf4db5da007fa3ea6ed95a3e27f0d828, deletion_comment = False)

    "administratorLoginPassword": {

Can you annotate it with x-ms-secret?
https://github.com/Azure/autorest/blob/main/docs/extensions/readme.md#x-ms-secret

Refers to: specification/sql/resource-manager/Microsoft.Sql/preview/2021-11-01-preview/Servers.json:897 in b3060af. [](commit_id = b3060afadf4db5da007fa3ea6ed95a3e27f0d828, deletion_comment = False)

    "publicBlob": {

Is this not a secret?

Refers to: specification/sql/resource-manager/Microsoft.Sql/preview/2021-11-01-preview/ServerTrustCertificates.json:240 in b3060af. [](commit_id = b3060afadf4db5da007fa3ea6ed95a3e27f0d828, deletion_comment = False)

    "hubDatabasePassword": {

Can you annotate it with x-ms-secret?
https://github.com/Azure/autorest/blob/main/docs/extensions/readme.md#x-ms-secret

Refers to: specification/sql/resource-manager/Microsoft.Sql/preview/2021-11-01-preview/SyncGroups.json:947 in b3060af. [](commit_id = b3060afadf4db5da007fa3ea6ed95a3e27f0d828, deletion_comment = False)

{

Where is this example used? I couldn't locate the api..

Refers to: specification/sql/resource-manager/Microsoft.Sql/preview/2021-11-01-preview/examples/CreateDwDatabaseCrossSubscriptionPITR.json:1 in b3060af. [](commit_id = b3060afadf4db5da007fa3ea6ed95a3e27f0d828, deletion_comment = False)

{

Where is this example used? I couldn't locate the api..
Refers to: specification/sql/resource-manager/Microsoft.Sql/preview/2021-11-01-preview/examples/CreateDwDatabaseCrossSubscriptionPITR.json:1 in b3060af. [](commit_id = b3060af, deletion_comment = False)

here is the usage: https://github.com/Azure/azure-rest-api-specs/pull/18005#discussion_r820031854

    "hubDatabasePassword": {

Can you annotate it with x-ms-secret?
https://github.com/Azure/autorest/blob/main/docs/extensions/readme.md#x-ms-secret
Refers to: specification/sql/resource-manager/Microsoft.Sql/preview/2021-11-01-preview/SyncGroups.json:947 in b3060af. [](commit_id = b3060af, deletion_comment = False)

@j5lim addressed your x-ms-secret comments in 7d092f0b534e9e23ed93ce0793cb8ddf08be6168

@ericshape , can you resolve conflicts?

/azp run

@raych1 do you know why the pipeline is not triggered?

we ignore these errors due to the existing legacy specs and example files cannot be deleted.

Rule
Message

❌ UNREFERENCED_JSON_FILE
The example JSON file is not referenced from the swagger file.readme: specification/sql/resource-manager/readme.mdjson: stable/2014-04-01-legacy/examples/ReplicationLinkDelete.json

❌ UNREFERENCED_JSON_FILE
The example JSON file is not referenced from the swagger file.readme: specification/sql/resource-manager/readme.mdjson: stable/2014-04-01-legacy/examples/ReplicationLinkFailover.json

❌ UNREFERENCED_JSON_FILE
The example JSON file is not referenced from the swagger file.readme: specification/sql/resource-manager/readme.mdjson: stable/2014-04-01-legacy/examples/ReplicationLinkUnlink.json

❌ UNREFERENCED_JSON_FILE
The swagger JSON file is not referenced from the readme file.readme: specification/sql/resource-manager/readme.mdjson: Microsoft.Sql/stable/2014-04-01-legacy/replicationLinks_legacy.json

❌ MISSING_APIS_IN_DEFAULT_TAG
The default tag does not contain all APIs in this RP. Please make sure the missing API swaggers are in the default tag.readme: specification/sql/resource-manager/readme.mdjson: Microsoft.Sql/stable/2014-04-01-legacy/replicationLinks_legacy.json

Rule	Message
❌ UNREFERENCED_JSON_FILE	The example JSON file is not referenced from the swagger file.
readme: specification/sql/resource-manager/readme.md
json: stable/2014-04-01-legacy/examples/ReplicationLinkDelete.json
❌ UNREFERENCED_JSON_FILE	The example JSON file is not referenced from the swagger file.
readme: specification/sql/resource-manager/readme.md
json: stable/2014-04-01-legacy/examples/ReplicationLinkFailover.json
❌ UNREFERENCED_JSON_FILE	The example JSON file is not referenced from the swagger file.
readme: specification/sql/resource-manager/readme.md
json: stable/2014-04-01-legacy/examples/ReplicationLinkUnlink.json
❌ UNREFERENCED_JSON_FILE	The swagger JSON file is not referenced from the readme file.
readme: specification/sql/resource-manager/readme.md
json: Microsoft.Sql/stable/2014-04-01-legacy/replicationLinks_legacy.json
❌ MISSING_APIS_IN_DEFAULT_TAG	The default tag does not contain all APIs in this RP. Please make sure the missing API swaggers are in the default tag.
readme: specification/sql/resource-manager/readme.md
json: Microsoft.Sql/stable/2014-04-01-legacy/replicationLinks_legacy.json


Need to escape illegal characters in XML

These are some HTML type escapes that may work:
xmlize = {}
xmlize[">"] = ">"
xmlize["<"] = "<"
xmlize["\n"] = ""

Added escapes for all XML illegal characters in XML_Base class


[QUESTION] Field mask

First check

[x] I used the GitHub search to find a similar issue and didn't find it.
[x] I searched the FastAPI documentation, with the integrated search.
[x] I already searched in Google "How to X in FastAPI" and didn't find any information.

Description
Does has fast api a field masks like a flask-rest-plus https://flask-restplus.readthedocs.io/en/stable/mask.html ?

Thanks for the help here @dmontagu ! :cake: :bowing_man:
And thanks @Hedgehogues for reporting back and closing the issue :+1:


Add negative caching tests and fixes.

This adds test coverage for the negative caching feature and makes some
fixes as a result of the test's findings.

Cherry-picked to 8.1.x by #7359

Cherry-picked to 8.1.x by #7359


Bug: ApiParam and collection format

implementation "io.springfox:springfox-boot-starter:3.0.0"
Spring boot version: 2.3.1.RELEASE
      @ApiParam(
          value = "some description",
          required = false,
          allowMultiple = true,
          collectionFormat = "csv"
      )
      val somename: List<String>?

Becomes
{
  name: "somename",
  in: "query",
  description: "some description",
  required: false,
  type: "array",
  items: {
    type: "string"
  },
  collectionFormat: "**multi**"
}

It looks to be correct in ApiParamParameterBuilder, but is turned to multi by the time it reaches RequestParameterMapper

It should preserve the override... thanks for reporting

I think I have a fix, not sure if it's the correct way to do it though

Its the right fix! thank you!


Introduce ss:ips RTP in the run's global namespace

Very often the application developer has to know the IP addresses of all deployment VMs. As of now we could find them out by parsing the ss:groups1 , then the node's <nodename>:ips and finally the <nodename>.<id>:hostame.
Since this is a fairly common need, we should be able to provide a direct way to get all IP addresses with e.g. ss:ips. Arguably we could provide more run details in an easily parsable json-like object GETable at once.

1 There is currently a bug preventing GETing the ss:groups though: https://github.com/slipstream/SlipStreamServer/issues/627

For Cyclone applications with networking services, it would be common that the application developer has to know which component instances . compose the deployment.
Getting their IP addresses is probably the most common use case for that, but being able to get the tags each instance has will be another interesting use case.
So IMHO it would be interesting to have a direct way to get all component instances in the deployment e.g. ss:instances returning a coma separated list of ..
With that set, either retrieving an instance run details from an easily parsable json-like object GET able at once, or retrieving each property individually have almost same value to me.


New Feature: Vector Shuffling

This is a back-port of a feature I created in my own fork which now has heavy refactors, but it was working so well, I felt I had to clean it up and offer it back to the main repo.
I've been working a lot with multi-vector embeddings lately.
One issue I've been having is what I had termed as "vector cross-talk".  This is where you might be training an embedding on a character with blue eyes and red hair, but the AI sometimes generates imagery of the character with red eyes and blue hair, or blue eyes and blue hair, etc.
I suspected that the cross-talk is because vectors are encoding individual colors represented by the character and the AI sometimes just free associates them to other features of the embedding and/or prompt.  So, what if the AI could not rely on vector ordering as much when learning?
This introduces a feature called vector shuffling, where the embeddings of individual vectors are shuffled randomly when they're inserted into the prompt during training.  The idea is that it will force it to encode sharper and more focused concepts into each vector.  No more blue AND eyes AND red AND hair.  Because it's scrambled, it can only learn the concept correctly if it encodes the entirety of the concept in one embedding, IE blue eyes AND red hair.
This is probably why using only 1 vector got the best results in the study; it had no choice but to encode a focused embedding when it had only 1 vector.  This feature tries to bridge the gap and give the best of both worlds: more storage for more complex subjects and focused vectors that resist leaking into other aspects of the prompt.
That's the theory, anyways.  I'm no data scientist, so someone who is can take this and test these claims, but doing this seemed to cut down on vector cross-talk in my tests.  It can still happen, but it significantly reduced occurrences and their severity when using the trained embedding.
You can enable this feature by setting model.params.personalization_config.params.shuffle_mode to true in your project's config YAML file, and it will shuffle all vectors.
Having done a lot of experimentation with the concept, it also supports the following additional options.  Experiment with them and see what works best for your particular subject.

all or on or true - Shuffles all vectors.
off or false - Disables shuffling; the default if it is not specified in your config.
trailing - With 3 or more vectors, shuffle all vectors after the first.  This provides a stable "intro" vector.
leading - With 3 or more vectors, shuffle all vectors before the last.  This provides a stable "outro" vector.
between - With 4 or more vectors, shuffles all vectors between the first and last.

This creates stable "intro" and "outro" vectors, allowing the embedding to take some advantage of token ordering during training.  That is, if you use your embedding like a name, this preserves a chance of learning that intended usage and taking advantage of that.  The idea is it will learn to use the first and last vectors as cues leading in and out of the embedding cleanly and minimize disruption to other parts of the prompt.

progressive - A special mode for progressive_words.  Like between, it also establishes stable "intro" and "outro" vectors, but ensures that the first and last vectors are kept the same throughout training as more vectors are added.

Needs num_vectors_per_token to be at least 3 to have any notable effect.
Vectors between the first and last are still shuffled as in between, when there's enough vectors unlocked to do so.
You will want to make sure that all vectors get some training (num_vectors_per_token * 2000 steps of training need to have occurred), otherwise some of the middle vectors may still be the initialization word's embedding.

dynamic - The "just make it work no matter what" option that favors stability.  This tries to always shuffle the vectors but also tries to establish stable intro and outro vectors when the number of vectors permits it.  What shuffle mode it ultimately uses differs based on how many vectors there are.

With 1 vector, uses off.
With 2 vectors, uses all.
With 3 vectors, uses trailing to at least establish a stable intro vector.
With 4 or more vectors, uses between to establish both intro and outro vectors.

If the number of vectors is below the supported number for an option, it acts the same as off unless otherwise noted.
The dynamic setting exists because some of my experiments allowed for different numbers of vectors for different placeholders when training multi-placeholder embeddings (like in the per_image_tokens mode).  It's also just a good option to suggest when someone isn't sure which to use.  It will try to provide as much benefit as it can regardless of what num_vectors_per_token is set to.

@TaleirOfDeynai Hi, i like your idea. Could your provide any example images with / without your training strategy? And did you compare between with all shuffling modes for large number of vectors (4+ for example)?

@TaleirOfDeynai I've tested the idea with vector shuffling and does sometimes improve the results for me, but at the same time leads to multiple people being generated lol (i'm using all shuffling mode with 4 vectors). The images below are generated with prompt: <placeholder-token> portrait
without shuffling:

With shuffling (notice there are almost always multiple persons generated).

+1 for this idea. For me random shuffling of all vectors with probability of 1 gives the best results.

This is very interesting. It seems like diffusion models might be better with a bag of words like approach.

this looks promising and should be implemented i think


Make UI Translateable

It would be great if this app could be translated. When that's made possible, I'll provide the Dutch translation. :-)

And I will translate to russian

Translation support is here (Implemented using gettext)! https://github.com/nlogozzo/NickvisionTagger/releases/tag/2022.11.0-beta1

Please submit a PR with your translations by Nov 10, 2022 to be included in the V2022.11.0 stable release for this week.

Thank you everyone for your help with translating this application, making it accessible to more people than ever before. A stable release will be released this week by Nov 10, 2022 with your translations :)
I'm closing this issue as completed since the translation infrastructure is complete.


Crackle.com single video "MPD 403 Forbidden" and downloads 1GB non-HD video

Checklist

[X] I'm reporting a broken site support
[X] I've verified that I'm running youtube-dl version 2019.09.28
[X] I've checked that all provided URLs are alive and playable in a browser
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones

Verbose log

youtube-dl -v https://www.sonycrackle.com/watch/972/2498714
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.sonycrackle.com/watch/972/2498714']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.09.28
[debug] Python version 2.7.15+ (CPython) - Linux-4.15.0-64-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.6, ffprobe 3.4.6, rtmpdump 2.4
[debug] Proxy map: {}
[Crackle] 2498714: Downloading media JSON as US
[Crackle] 2498714: Downloading m3u8 information
[Crackle] 2498714: Downloading MPD manifest
WARNING: Failed to download MPD manifest: HTTP Error 403: Forbidden
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on u'http://content-ausw3.uplynk.com/4d55bdb9330e422cbd796ef413b2f1d2/h.m3u8?expand=117c&ad=crackle_resp_web_live&ad.locationDesc=crackle_responsive_web_up_us_shows&ad.bumper=&ad.preroll=1&extsid=1&ad.metr=7&euid=&exp=1570311637&ct=a&cid=4d55bdb9330e422cbd796ef413b2f1d2&sig=7341c1cf7fd5ddc4b86b24cfffd8a49e6db5c02af94191ffaa3cbdbda84f99b8&pbs=e9dac3ec981b4c2d8928a7ed49c931ee'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 330 (not including 161 ad)
[download] Destination: A Police Officer And A Gentleman-2498714.mp4
[download]   0.4% of ~964.35MiB at  5.57MiB/s ETA 06:35

Description

Every episode of this series allowed the MPD to download and resulted in around a 150mb file as this is a 90s show that was not HD to the best of my research tells me.
So the fact that this episode randomly downloads almost a 1GB file is unknown. The resulting file does play and look right, but specs wise is not HD but 640x480 just like it's sister files.

This is what they serve. All blames go there.


Adding type definitions for dts-bundle

DtsBundle is for bundling type definitions to be consumed by other
libraries.

@pspeter3 thanks mate!
Cc: @Bartvds


Unicode indexing error?

Here's an example notebook which fails:

title: Unit Checked Arithmetic via Unitful.jl
author: Chris Rackauckas
Units and dimensional analysis are standard tools across the sciences for checking the correctness of your equation. However, most ODE solvers only allow for the equation to be in dimensionless form, leaving it up to the user to both convert the equation to a dimensionless form, punch in the equations, and hopefully not make an error along the way.
DifferentialEquations.jl allows for one to use Unitful.jl to have unit-checked arithmetic natively in the solvers. Given the dispatch implementation of the Unitful, this has little overhead.
Using Unitful
To use Unitful, you need to have the package installed. Then you can add units to your variables. For example:
using Unitful
t = 1.0u"s"

Notice that t is a variable with units in seconds. If we make another value with seconds, they can add
t2 = 1.02u"s"
t+t2

and they can multiply:
t*t2

You can even do rational roots:
sqrt(t)

Many operations work. These operations will check to make sure units are correct, and will throw an error for incorrect operations:
t + sqrt(t)

Using Unitful with DifferentialEquations.jl
Just like with other number systems, you can choose the units for your numbers by simply specifying the units of the initial condition and the timestep. For example, to solve the linear ODE where the variable has units of Newton's and t is in Seconds, we would use:
using DifferentialEquations
f = (y,p,t) -> 0.5*y
u0 = 1.5u"N"
prob = ODEProblem(f,u0,(0.0u"s",1.0u"s"))
sol = solve(prob,Tsit5())

Notice that we recieved a unit mismatch error. This is correctly so! Remember that for an ODE:
\frac{dy}{dt} = f(t,y)

we must have that f is a rate, i.e. f is a change in y per unit time. So we need to fix the units of f in our example to be N/s. Notice that we then do not receive an error if we do the following:
f = (y,p,t) -> 0.5*y/3.0u"s"
prob = ODEProblem(f,u0,(0.0u"s",1.0u"s"))
sol = solve(prob,Tsit5())

This gives a a normal solution object. Notice that the values are all with the correct units:
print(sol[:])

We can plot the solution using the plot recipe.
#Pkg.clone("https://github.com/ajkeller34/UnitfulPlots.jl")
using UnitfulPlots, Plots
gr()
plot(sol.t,sol[:],lw=3)

Notice that here we pulled the units for the label directly from the solution. Thus if the units change, the labels will change automatically.

The error is:
StringIndexError("Unitful.Quantity{Float64,�*�*�^-2,Unitful.FreeUnits{(N,),�*�*�^-2,nothing}}[1.5 N, 1.53621 N, 1.77204 N]", 75)
string_index_err(::String, ::Int64) at string.jl:12
Type at substring.jl:32 [inlined]
Type at substring.jl:38 [inlined]
Type at substring.jl:44 [inlined]
Type at substring.jl:40 [inlined]
getindex(::SubString{String}, ::UnitRange{Int64}) at substring.jl:204
wrapline(::SubString{String}, ::Int64) at format.jl:322
wraplines(::String, ::Int64) at format.jl:310
format_chunk(::Weave.CodeChunk, ::Dict{Symbol,Any}, ::Weave.JMarkdown2HTML) at format.jl:186
format(::Weave.WeaveDoc) at format.jl:33
#weave#16(::String, ::Symbol, ::String, ::Dict{Symbol,String}, ::Symbol, ::String, ::Nothing, ::String, ::Symbol, ::Bool, ::Nothing, ::Nothing, ::Nothing, ::Array{String,1}, ::String, ::typeof(Weave.weave), ::String) at Weave.jl:113
#weave at none:0 [inlined]
weave_file(::String, ::String, ::Tuple{Symbol,Symbol,Symbol}) at DiffEqTutorials.jl:21
weave_file(::String, ::String) at DiffEqTutorials.jl:8
top-level scope at none:0

It looks like an error with indexing unicode incorrectly, but is hard to isolate.

Thanks for reporting! There seems to be a bug in wrapping the output https://github.com/mpastell/Weave.jl/blob/1e630ed825e8c4c696e5d3fd4fe571a19b892511/src/format.jl#L319, you can avoid it by using wrap=false.

I'm getting unsupported kwarg for wrap.

weave(tmp,doctype = "md2html",out_path=dir,args=args, wrap=false)

You need to use it as a chunk option ```julia; wrap=false . See: http://weavejl.mpastell.com/stable/chunk_options/

Can there be a global way of disabling the wrap? I don't want the wrapping anyways. It makes the html outputs look weird. Example: https://htmlpreview.github.io/?https://github.com/JuliaDiffEq/DiffEqTutorials.jl/blob/master/html/number_types.html

wrap=false fixes the html output, but the PDF output still is having issues.

You can use set_chunk_defaults function to set global defaults, it would make sense to allow passing default options directly to weave if you'd have time to add the feature.

OK, this should now be fixed.

You can now set default options in document header: http://weavejl.mpastell.com/dev/chunk_options/#Set-default-chunk-options-1


Fix caching with prereleases

This PR: fetch latest stable release or fetch last 20 releases to find a non-draft prerelease.
For now, cache system fetches release with https://api.github.com/repos/${repo}/releases?per_page=1. And there is 2 problems with that:

In non-PRE mode: It returns a release but this release can be a draft or a prerelease. Using releases/latest prevents this.
In PREmode: If the last release is a stable release, it doesn't return latest prerelease

Fetching 20 releases is completely arbitrary, but seems enough. No need to scan all releases with pagination system for prerelease. I can maybe put it in an ENV variable to tweak it but it seems over-engineered

Let's always load as highest number of releases (I think it's a 100), because it might be the case that the "latest" release is a pre-release. That means when PRE is off, it wouldn't work.

It works because it uses another API endpoint releases/latest
You'll always have the last non-draft stable release

BTW: I think it could be useful that PRE mode returns stable release if it is newer than last prerelease. It it doesn't, it force maintainers to publish a pre-release for each stable release if they don't want that their users get stuck in old pre-release version.


time span of articles

hi, yihui. I am a reader of your blog.
I find your blog starts from your fourth year of undergraduate education. Do you have ever considered put your diaries before that time in your blog? I have tried to do so recently and find some parts are very tedious because old articles are in microsoft word format. The older are scanned version of notebooks. You can check my github pages if you want and any comments on this issue is welcome.

我上大学后没写多少日记，可能用笔写过几篇，但现在应该是找不到了。高中及以前的日记本应该也都散失了。我现在能做的就是有空时凭记忆把那时候的一些事再写出来。
你的旧 Word 文档可以用 Pandoc 转成 Markdown。扫描的日记本不妨就直接放图片嘛，非要整理的话，也许有光学识别的软件可用。

Thanks for your suggestion. pandoc works for docx. My old documents are mostly in doc format, which is different from docx. As far as know, there is some software which can convert doc to docx in batch mode.


After upgrade HA to 2024.8.0 integration stop

After the HA update, the integration stopped working correctly despite there being no failure logs or recognition of devices that previously worked normally.
anything to do to fix this bug before the next update?

I'm getting the same thing. Entities show as unavailable when running 2024.8.0.  Downgrading was a mess.  I had to reinstall the integration, and that triggered LG to require a new password because I hadn't changed it in 30 months. (I'm glad LG is so on top of security. Sometimes I can't sleep at night because I worry that somebody is trying to break into my washing machine.)  It took an hour or so for LG to accept my new password from HA.

I tried again this morning and the LG integration loaded normally on the first try, so I don't think the issue is due to 2024.8.0, but more likely due to issue 772 or the LG password drama I had yesterday.

I tried again this morning and the LG integration loaded normally on the first try, so I don't think the issue is due to 2024.8.0, but more likely due to issue 772 or the LG password drama I had yesterday.

Your device is also a washing machine as you mentioned above, here I still have no communication with mine, I will wait a little longer to see if any developer says something, for now I took a look at the logs and guess what, everything is fine without any errors.

I actually have 4 thinq devices--washer, dryer, dishwasher, and refrigerator, but they're all or nothing.
Have you tried logging out of the LG website on your phone and logging back in? Maybe you're facing the same required password change that others have seen.

I just did this but without success, I'm going to try again because the LG account I use is not the main LG account but a secondary account, I'll test it here and then submit the result here.

To conclude the matter, I checked and the problem was indeed in the LG cloud, after deleting and re-registering my device it started working normally again.


Please add example of Play integration as standalone project

It is not obvious how to set up a Play project to use scalajs-transport (without Akka), and how to specify the dependencies. Please add a standalone example on how to do this.

I think examples/rpc does just that (the build declaration is [here]https://github.com/OlivierBlanvillain/scalajs-transport/blob/master/build.sbt#L218), it does not depends on Akka). Do think I should provide an example using neither Autowire nor Akka? Or are you referring to the fact that the build file uses dependsOn instead of libraryDependencies?

Yes, it has an internal dependency instead of to a Maven repo. But I found some hints in the buildfile of the game.


[Feature]:  ROCm Using shared memory When GPU runs out of memory

Suggestion Description
When I paint with ComfyUI, if the workflow is more complex, more models need to be loaded, or larger images are generated, there is a HIP out of memory, even if there is still enough space in shared memory at that point.
But when I switch to windows and run ComfyUI with zluda, I find that when the GPU runs out of memory, it will use shared memory to continue running instead of throwing a HIP out of memory, I would like to know if there is a plan to support the use of shared memory in subsequent releases.
Operating System
Ubuntu
GPU
RX 7900xtx
ROCm Component
ROCm 6.2.0

Hi @void-Man, are you able to consistently reproduce this issue with a specific model? If so, could you please share the steps to reproduce the HIP out of memory error?

I have the same issue, I have 64 GB system ram that sits there unused :(

the same issue:
System:
CPU AMD Ryzen 9950X
RAM 128 GB DDR5
GPU0 AMD Radeon PRO W7900
GPU1 AMD Radeon RX7900XTX
ROCM: 6.3.1
Ubuntu 24.04 LTS (currently patched)
In a multi-GPU setup, only the VRAM of one graphics card is used, the 128GB are not used by ollama (with ROCm) and only an error message: Out of Memory is given.