updating to Microsoft.NET.Test.Sdk 16.9.1 causes ReflectionTypeLoadException

TFM: net48 and net5
Repro: https://github.com/SimonCropp/TestSdkRepro

    [Fact]
    public void Foo()
    {
        foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
        {
            foreach (var type in assembly.GetTypes())
            {
                Debug.WriteLine(type);
            }
        }
    }

run dotnet test
Result: ReflectionTypeLoadException

Test run for C:\Code\TestSdkRepro\bin\Debug\net5.0\TestSdkRepro.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:00.79]     UnitTest1.Foo [FAIL]
  Failed UnitTest1.Foo [96 ms]
  Error Message:
   System.Reflection.ReflectionTypeLoadException : Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.VisualStudio.Coverage.CoreLib.Net, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
  Stack Trace:
     at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at System.Reflection.Assembly.GetTypes()
   at UnitTest1.Foo() in C:\Code\TestSdkRepro\UnitTest1.cs:line 12

Also seen in other projects. https://github.com/SwissLife-OSS/snapshooter/pull/126 note they worked around it by catching ReflectionTypeLoadException
Environment:

.net sdk 5.0.103 and 5.0.200-preview2
Windows Server 2016

interestingly (random guess). adding a ref manual ref to Microsoft.VisualStudio.Coverage.CoreLib.Net.dll make the test pass
  <ItemGroup>
    <Reference Include="Microsoft.VisualStudio.Coverage.CoreLib.Net">
      <HintPath>D:\nugetglobalpackages\microsoft.codecoverage\16.9.1\build\netstandard1.0\Microsoft.VisualStudio.Coverage.CoreLib.Net.dll</HintPath>
    </Reference>
  </ItemGroup>

not sure if that helps anyone debugging?

The excepetion is raised by loading Microsoft.TestPlatform.Utilities.
I guess the problem is that in the testhost.deps.json file inside the nuget package Microsoft.TestPlatform.Utilities is referencing Microsoft.VisualStudio.Coverage.CoreLib.Net/16.9.0-beta.21119.4.
https://github.com/microsoft/vstest/blob/6a858a323caa439115e9e3cf1c958b04f4ff9993/scripts/build/TestPlatform.Dependencies.props#L35

@Haplois probably connected to the other issue you are solving.

The issue here is, Microsoft.CodeCoverage package doesn't include all of its dependencies correctly thus Microsoft.VisualStudio.Coverage.CoreLib.Net.dll file is missing from the build output.
/cc: @fhnaseer @jakubch1

@fhnaseer could you please cherry pick change to load CoreLib.Net.dll dynamically to release branch 16.9?

@fhnaseer could you please cherry pick change to load CoreLib.Net.dll dynamically to release branch 16.9?

Please let me know when it's done; I also have a hotfix to include. We can release a preview when both are done.

Also, I just created the branch preview/16.9.2 please use that, after all is done we can merge to rel branch. @fhnaseer

is there an eta on 16.9.2? can i help by smoke testing a preview nuget?

Version 16.9.2-preview-20210317-01 does include this fix, and available on our daily feed.

@Haplois can that version be deployed to nuget?

@SimonCropp, we'll be releasing a preview this week. Probably on Wednesday.

Had same issue. Can confirm that upgrading to your fixed preview version solved the problem. Thanks for being so quick.

Closing, as have confirmation that it worked after update.


Remove the compliance history API tests

The tech preview feature is being removed so the tests are no longer needed.
Relates:
https://issues.redhat.com/browse/ACM-15291
https://issues.redhat.com/browse/ACM-13208

/test "test-e2e-grc-framework"

/test test-e2e-grc-framework

/retest


Upgrade package dependencies

Upgrade dependencies and make sure the code still works

@Aaronontheweb I'm a little bit stuck, if you have some time and could have a look if you find something please?
There are 3 failing tests:

The rest are ok:

If you debug at src\Akka.Persistence.Sql\Db\AkkaPersistenceDataConnectionFactory.cs inside the HasConversion funcs (you can temporarly use HasConversionFunc if the breakpoints don't get hit), you will see the DateTime gets converted into Ticks when the test prepares its snapshot data and writes it to MS Sqlite.
if (config.ProviderName.ToLower().Contains("sqlite") ||
    config.ProviderName.ToLower().Contains("postgresql"))
{
    builder
        .Member(r => r.Created)
        .HasDataType(DataType.Int64)
        .HasConversion(
            r => r.Ticks,
            r => new DateTime(r));
}

However, when it tries to read them, it fails with:
Got a message of the expected type <Akka.Persistence.LoadSnapshotResult>. Also expected the predicate to return true but the message {LoadSnapshotResult<toSeqNr:<PHONE_NUMBER>854775807, snapshot: >} of type <Akka.Persistence.LoadSnapshotResult> did not match
Expected: True
Actual:   False
   at Akka.TestKit.Xunit2.XunitAssertions.AssertTrue(Boolean condition, String format, Object[] args)
   at Akka.TestKit.TestKitBase.AssertPredicateIsTrueForMessage[T](Predicate`1 isMessage, T m, String hint)
   at Akka.TestKit.TestKitBase.<>c__DisplayClass121_0`1.<ExpectMsgAsync>b__0(T m, IActorRef sender)
   at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelopeAsync[T](Nullable`1 timeout, Action`2 assert, String hint, CancellationToken cancellationToken, Boolean shouldLog)
   at Akka.TestKit.TestKitBase.InternalExpectMsgAsync[T](Nullable`1 timeout, Action`2 assert, String hint, CancellationToken cancellationToken)
   at Akka.TestKit.TestKitBase.ExpectMsgAsync[T](Predicate`1 isMessage, Nullable`1 timeout, String hint, CancellationToken cancellationToken)
   at Akka.TestKit.TestKitBase.ExpectMsg[T](Predicate`1 isMessage, Nullable`1 timeout, String hint, CancellationToken cancellationToken)
   at Akka.Persistence.TCK.Snapshot.SnapshotStoreSpec.SnapshotStore_should_delete_all_snapshots_matching_upper_sequence_number_and_timestamp_bounds()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

If you debug src\Akka.Persistence.Sql\Snapshot\ByteArraySnapshotDao.cs at SnapshotForMaxSequenceNrAndMaxTimestamp you will see the values being passed into it are fine, however the FirstOrDefaultAsync returns null, even though the row is there:

I currently have no idea yet why this is happening. I was trying to find a way to debug the exact query being sent to Sqlite, maybe it doesn't convert the timestamp parameter to ticks properly?

Of course, by typing out my problem I get a step closer to the answer :) I managed to find the query it sends to the database:
SELECT
	[r].[persistence_id],
	[r].[sequence_number],
	[r].[created],
	[r].[snapshot],
	[r].[manifest],
	[r].[serializer_id]
FROM
	[snapshot] [r]
WHERE
	[r].[persistence_id] = @persistenceId AND [r].[sequence_number] <= @sequenceNr AND
	DateTime([r].[created]) <= DateTime(@timestamp_1)
ORDER BY
	[r].[sequence_number] DESC
LIMIT @take

That DateTime([r].[created]) <= DateTime(@timestamp_1) is probably the problem, I'll try to figure out why it does that and not simply does the comparison with ticks.

@CumpsD I'll look into it

Just checked with the dev branch, the query there is:
SELECT
	[r].[persistence_id],
	[r].[sequence_number],
	[r].[created],
	[r].[snapshot],
	[r].[manifest],
	[r].[serializer_id]
FROM
	[snapshot] [r]
WHERE
	[r].[persistence_id] = @persistenceId AND [r].[sequence_number] <= @sequenceNr AND
	[r].[created] <= @timestamp_1
ORDER BY
	[r].[sequence_number] DESC
LIMIT @take

I'll have a look at linq2db, I'm guessing something changed in the breaking changes from 3->5. I've read all the release notes though, guess I missed something

3.6.0 broke it, https://github.com/linq2db/linq2db/wiki/Release-Notes-3.0.0#release-360

3.5.2 is the last version the test succeeds.
3.6.0 broke it
Release notes: https://github.com/linq2db/linq2db/wiki/Release-Notes-3.0.0#release-360
PR: https://github.com/linq2db/linq2db/pull/3343
I'll have a look at what changed

Another tidbit of information: https://github.com/linq2db/linq2db/issues/2432

For some reason, the azure checks don't run anymore

I’ll need to fix that - repo name change broke itSent from my iPhoneOn Mar 21, 2023, at 4:05 AM, David Cumps @.***> wrote:
For some reason, the azure checks don't run anymore
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

Just linking some for admin purposes.

Fixes #162
Fixes #161
Fixes #160
Fixes #139
Fixes #113
Fixes #112
Fixes #109
Fixes #46
Fixes #43


Merge contrib/go into master, minor syntax fixes

This includes:

Rebase of go into master, with conflicts resolved
Fix tensor_c_api.h to resolve Go import failures

Can one of the admins verify this patch?

I saw your other change doing this, so I'm sorry for the churn, but would you mind rebasing master on top of the go branch instead? For various reasons (not working with Jenkins, API changes, etc), we'd prefer not to merge it into master at the moment.

Sure, no worries at all. I will close this and rebase master -> go sometime tomorrow.


Show button

closes #312
Add show button to the file input page. If a user clicks on that button, results for each result type (if they exists) are shown.

Coverage decreased (-0.11%) to 45.47% when pulling cc0ec50e7d2d5d42bb9b9b479d74f850d34fa926 on show_button into 9c95c6f9c425a0f10415ebb87d566684255af23a on master.


Support boxes overlap spec prose and each other

The support boxes have two problems with styling atm:

They overlap spec prose in narrow windows.
They overlap each other when sections are short.

See https://drafts.csswg.org/css-cascade-5/#default for an example
I suggest starting them out as collapsed rather than expanded, that should fix both problems.

I think this got fixed, by starting the boxes out collapsed as you suggested, although I don't see the commit that did it. I do see another problem at https://drafts.csswg.org/css-cascade-4/#initial, where the CanIUse box sits directly on the MDN box: https://github.com/tabatkins/bikeshed/issues/2442.


Rushdown enchantment crashes server

Recently members of my minecraft server have gotten ahold of the either incomplete or broken rushdown enchantment, and just by having an enchanted book of it it proceeded to cause the entire server to crash which was only resolved when we edited his player data to delete the book from his inventory.  Just wanted to log it here in case theres a way to prevent it from happening in the future or if you're unaware.

A crash log would be appreciated.

If there is no crash log, we cannot help you. Regardless, Rushdown should already be working completely fine by next release.


Typo "General Availability [(GA) announcements]"→"[General Availabili…

…ty(GA) announcements]"
https://docs.microsoft.com/en-us/azure/azure-sql/database/features-comparison

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Looks good; thanks.


Variable Response => Header does not find executed responses

Expected Behavior
I have two calls and would like to use the Response Header value from one call in the request Header of the other call.

The first call "Get Data Object" executed successful and I can see the ETag value.
If I now selected Response => Header and select the Get Data Object call and enter ETag I would expect the value to be copied from this response.
Actual Behavior

But when i use this request in a Response => Header variable the live preview complains that there are no responses for this call. It also does not work when I close this window or if i try to execute the call. Refreshing the preview via the refresh button also does nothing.
It does not find any of the successful responses.
Reproduction Steps
No response
Is there an existing issue for this?

[X] I have searched the issue tracker for this problem.

Additional Information
No response
Insomnia Version
2023.1.0
What operating system are you using?
Windows
Operating System Version
Windows 10 22H2
Installation method
winget
Last Known Working Insomnia version
No response

Ok with some digging in the insomnia Code i can now finally narrow the issue down.
If I do not use an Environment like this:

It does not work. It does not find any responses. As soon as I create an empty environment (besides the base environment) it works.

My best guess as a non-js dev was that this here has troubles with no specific environment:
packages\insomnia\src\models\response.ts
async function _findRecentForRequest(
  requestId: string,
  environmentId: string | null,
  limit: number,
) {
  const query: Query = {
    parentId: requestId,
  };

  // Filter responses by environment if setting is enabled
  if ((await models.settings.getOrCreate()).filterResponsesByEnv) {
    query.environmentId = environmentId;
  }

  return db.findMostRecentlyModified<Response>(type, query, limit);
}

I tried it with older versions and it seems to be that it has this issue for longer than a year. (This is when I stopped because I knew it was working around that time and therefore started looking for another cause of this issue).


neighborhood_ages_over_time

Need columns and rows transposed. Sandbox need to be able to filter by year.
Example:
https://docs.google.com/spreadsheets/d/1AlgRdm5UoKeb2EGBWDNbJzaxXEDyr9fLGybqUjcO_Pc/edit?usp=sharing
also, side note,  should this maybe be named neighborhood_voters_ages_over_time?

@ylee22 said she would take a look at this.

will attempt to address on Friday

@hassanshamim no luck yet, but can continue effort if you get busy w/ something else..

This is updated in the database now under table neighborhood_ages_transposed_uniq.csv.  Will add a Django model to reflect the new table/format.


The L2 normalization of features

Dear author,
I read the code and found that all the embeddings in the loss function are L2-normalized in the projection head, but regarding the dequeue_and_enqueue function, why are the features normalized again at L122 and L134 in trainer_contrastive.py?

Hi @ccccly, the normalization in L122 is essential since new features are obtained for segments that are not guaranteed to be normalized. The one in L134 is indeed redundant. I wrote this to keep the code more consistent, but anyway it will have no effect. Feel free to remove it.

Thx! @tfzhou, I got it now.


. net core3.1 changing the server port has no effect

Describe your question
I want to use mqttnet as the server to receive messages from the front end of the web program. Because I'm afraid that the port of mqttnet using HTTP will block the port, I want to set a port for mqttnet alone, but I find that when I set other ports, the front-end cannot connect to the server, and only when I set the port of HTTP (Port: 5000) can I connect. I would like to ask where I did not configure the above problems, or the web side can only use the HTTP port.
I use the API in the. .net core web program on the back end, and I use the Vue on the front end.
server Code :

Which project is your question related to?

Client
ManagedClient
MQTTnet.Server standalone
Server
Generic

or the web side can only use the HTTP port.

yes

Describe your question
I want to use mqttnet as the server to receive messages from the front end of the web program. Because I'm afraid that the port of mqttnet using HTTP will block the port, I want to set a port for mqttnet alone, but I find that when I set other ports, the front-end cannot connect to the server, and only when I set the port of HTTP (Port: 5000) can I connect. I would like to ask where I did not configure the above problems, or the web side can only use the HTTP port. I use the API in the. .net core web program on the back end, and I use the Vue on the front end. server Code :

Which project is your question related to?

Client
ManagedClient
MQTTnet.Server standalone
Server
Generic

请问前端是这么写的吗？
const client3=mqtt.connect("ws://xxx.xxx.xxx.xxx : 端口", {
log: console.log.bind(console),
keepalive: 30,
clientId: "meterAlarmRecord",
username: "admin",
password: "123456",
path: "/mqtt",
});
我给后端也设置为端口5000了，但是vue还是连不上。。求助大佬。

@mimimiaomiao1 Try to use:
mqttServer.WithDefaultEndpoint()
    .WithEndPointPort(20000);

@mimimiaomiao1 Try to use:
mqttServer.WithDefaultEndpoint()
    .WithEndPointPort(20000);

mqttServerOptionsBuilder.WithDefaultEndpoint();
mqttServerOptionsBuilder.WithDefaultEndpointPort(port);
Not effective

@mimimiaomiao1 Try to use:
mqttServer.WithDefaultEndpoint()
    .WithEndPointPort(20000);

mqttServerOptionsBuilder.WithDefaultEndpoint().WithDefaultEndpointPort(port);
It's not possible either


Allow specifying plugin name instead of plugin object in results_store.add() call

connection.results.add() should accept a String instead of current Object.
{name: 'blah'} works, but garbles the output.

It can't. If you change the API of logger so I can pass in an initial string instead of a plugin name, I'd happily update result_store.

log_if_level is essentially using implicit plugin.toString() call, so that shouldn't be a problem.

Are you sure about that? I'd be happy to be wrong.


Refactor ProxyHttpEventArgsBase to move throttled requests to pluginData

As we've introduced pluginData in ProxyEngine in #506, let's see if we can refactor ProxyHttpEventArgsBase and move throttled requests from its constructor into pluginData. Throttling belongs to plugins and shouldn't be explicitly exposed in the engine or event args.

Moving information about the throttled requests to pluginData wouldn't be natural. Right now, plugin data is tied to session (single request+response). On the other hand, throttled requests apply as long as proxy is running. As such, it would be clearer to:

Rename pluginData to sessionData, to clearly indicate that the data is tied to a single session
Add globalData (or choose a more suitable name), for storing data that span multiple sessions.

Thoughts @garrytrinder?

I agree with the approach.
In terms of naming, in the HTTP spec a request\response is referred to as an exchange, see https://www.rfc-editor.org/rfc/rfc2616#section-1.4
How about we use exchangeData, to store data for request/response exchanges and sessionData which refers to the Proxy session itself.

To be honest, without knowing the spec, if I just looked at exchangeData, I'd have no idea what it's referring to. So I suggest we consider a different name. As for sessionData, Titanium uses it for a request session, so I suggest we align with it, because if we use it for proxy session it'll be confusing for others who don't have this background.


Default to containerless mode

Fixes https://issues.redhat.com/browse/MTA-4217 and https://issues.redhat.com/browse/MTA-4219

@aufi I agree. I think that's a good thing to be aware of and look out for. May need to discuss some solutions next week, because I'm not 100% sure on a good way to solve that.


Not uploading files in linux

Weights and Biases version: current
Python version: 3.6
Operating System: linux

Description
Not uploading files in linux/tensorflow.
When I write tensorflow even logs into my wandb directory (sample code in http://github.com/lukas/models in tutorials/image/cifar10/cifar10_train.py.
Traceback (most recent call last):
  File "cifar10_train.py", line 131, in <module>
    tf.app.run()
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "cifar10_train.py", line 127, in main
    train()
  File "cifar10_train.py", line 119, in train
    mon_sess.run(train_op)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 671, in run
    run_metadata=run_metadata)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1156, in run
    run_metadata=run_metadata)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1240, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1312, in run
    run_metadata=run_metadata)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1076, in run
    return self._sess.run(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
KeyboardInterrupt

Fixed in 0.6.30


ai summarizing tool

Instead of doom scrolling on an article, an ai summarizes the article.

I'm not sure I follow you. Doom scrolling refers to infinitely scrolling lists of content. Not reading an article.
Most people decide if they are interested in reading an article by reading and stopping if they don't like it. How will an AI help with this process?

I get your point but there are a lot of things that aren't as important in an article that an ai can summarize. There are a lot of technology like this that you can even use to summarize any text.

Sure there are Machine Learning models that can do this. But what is the actual utility of it? How does it improve the user experience?

It will improve the user experience by not getting bored into reading a very long article


Purefb Array Total Capacity

In 2021, I had initiated discussion with Pure team to get some answers on how to get the Array's total capacity from Purefb Metrics.
The ask was to get this metric purefb_array_capacity_bytes to show the capacity of the array. In the meantime, I had created the static capacity timeseries metric for dashboards to fill the gap.
I want to re-initiate the follow up on this so that we can get the Array capacity dynamically instead of statically.
def __init__(self, fb):
    self.fb = fb
    self.capacity = GaugeMetricFamily(**'purefb_array_capacity_bytes',**
                                   **'FlashBlade total capacity in bytes',**
                                   labels=[])

Can you please provide some information on this?
Does this metric exist today? if not, is there a way to calculate this based off of current available metrics? If not, is there an API I can use to build this metric?

Hi,
We currently export this metric as
purefb_array_space_bytes{space="capacity",type="array"} 8.9937250028093e+13
Please let me know if this fits your use case.


does Gradient checkpointing support multi-gpu ?

https://github.com/huggingface/transformers/pull/4659

The answer is yes, but you need two things (pytroch constraints):

use find_unused_parameters=False
make sure that all model parameters contribute to the calculation of the loss.

Here's an example
import os
import torch
from transformers import BertForPreTraining
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
torch.distributed.init_process_group(backend="nccl", rank=0, world_size=1)

model = BertForPreTraining.from_pretrained('bert-base-uncased').cuda()
model = torch.nn.parallel.DistributedDataParallel(model, find_unused_parameters=True)
outputs = model(torch.tensor([[1, 2, 3]]).cuda())
outputs[0].sum().backward()

# check parameters with no grad
for n, p in model.named_parameters():
    if p.grad is None and p.requires_grad is True:
        print('Parameter not used:', n, p.shape)  # prints unused parameters. Remove them from your model

outputs = model(torch.tensor([[1, 2, 3]]).cuda())
outputs[0].sum().backward()  # if any unused parameters remain in your model, this will break.

Hi,
I tried multi-gpu on longformer with a similar setup, and I found some layers that were unused. How do I correct this?
No forward parameters: module.longformer.pooler.dense.weight torch.Size([768, 768])
No forward parameters: module.longformer.pooler.dense.bias torch.Size([768])

would del p delete them from the model correctly?

Unfortunately that did not make a difference

how about p.requires_grad = False? the other solution is to change the model implementation and remove that parameter from the code.

Haven't read through the linked thread - but a quick workaround-hack I've employed before adds something like this to forward:
for p in self.parameters():
    loss += 0.0 * p.sum()

This ensures all parameters have a gradient.  If you know the extra parameters that are causing the problem you can restrict the loop to just those.

how about p.requires_grad = False? the other solution is to change the model implementation and remove that parameter from the code.

No luck either. Here is part of my implementation, in case I did it wrong:
            b_input_ids = batch[0].cuda(gpu)
            b_input_mask = batch[1].cuda(gpu)
            b_labels = batch[2].cuda(gpu)

            model.zero_grad()

            loss, logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

            loss = loss.mean()
            total_train_loss += loss.item()

            loss.backward()

            # check parameters with no grad
            for n, p in model.named_parameters():
                if p.grad is None and p.requires_grad is True:
                    print('No forward parameters:', n, p.shape)
                    p.requires_grad = False

            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            optimizer.step()

            scheduler.step()

Haven't read through the linked thread - but a quick workaround-hack I've employed before adds something like this to forward:
for p in self.parameters():
    loss += 0.0 * p.sum()

This ensures all parameters have a gradient. If you know the extra parameters that are causing the problem you can restrict the loop to just those.

Where would I need to implement this in the training loop?

In your code right above, add it after the line loss = loss.mean() so it looks like:
loss = loss.mean()
for p in model.parameters():
    loss += 0.0 * p.sum()

In your code right above, add it after the line loss = loss.mean() so it looks like:
loss = loss.mean()
for p in model.parameters():
    loss += 0.0 * p.sum()

I tried it and I think it caused some sort of memory access error. It wasn't able to reach the loss.backward() step. The stack trace is below:
/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
Traceback (most recent call last):
  File "ddp.py", line 202, in <module>
    main()
  File "ddp.py", line 57, in main
    train_model(0, train_dataset = Train_dataset)
  File "ddp.py", line 171, in train_model
    loss.backward()
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.3) Incorrect unused parameter detection. The return value of the `forward` function is inspected by the distributed data parallel wrapper to figure out if any of the module's parameters went unused. For unused parameters, DDP would not expect gradients from then. However, if an unused parameter becomes part of the autograd graph at a later point in time (e.g., in a reentrant backward when using `checkpoint`), the gradient will show up unexpectedly. If all parameters in the model participate in the backward pass, you can disable unused parameter detection by passing the keyword argument `find_unused_parameters=False` to `torch.nn.parallel.DistributedDataParallel`.
Exception raised from mark_variable_ready at /pytorch/torch/csrc/distributed/c10d/reducer.cpp:452 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f77e524b1e2 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10d::Reducer::mark_variable_ready(c10d::Reducer::VariableIndex) + 0x909 (0x7f76d65b7239 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #2: c10d::Reducer::autograd_hook(c10d::Reducer::VariableIndex) + 0xef (0x7f76d65b950f in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #3: <unknown function> + 0xa90676 (0x7f76d65b9676 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0xa95146 (0x7f76d65be146 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x4dd (0x7f76c8cd84dd in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #6: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f76c8cd9fa1 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #7: torch::autograd::Engine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x25c (0x7f76c8cd759c in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #8: torch::autograd::python::PythonEngine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x3c (0x7f76d60568dc in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #9: torch::autograd::Engine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0x803 (0x7f76c8cd69f3 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #10: torch::autograd::python::PythonEngine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0x4e (0x7f76d60566de in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #11: THPEngine_run_backward(THPEngine*, _object*, _object*) + 0xa54 (0x7f76d60573c4 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #29: torch::autograd::PyNode::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x183 (0x7f76d605e483 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #30: <unknown function> + 0x3375bb7 (0x7f76c8cddbb7 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #31: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7f76c8cd9400 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #32: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f76c8cd9fa1 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #33: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7f76c8cd2119 in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #34: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7f76d60564ba in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #35: <unknown function> + 0xc819d (0x7f76d749119d in /home/ec2-user/anaconda3/envs/JupyterSystemEnv/bin/../lib/libstdc++.so.6)
frame #36: <unknown function> + 0x7e75 (0x7f77f0a93e75 in /lib64/libpthread.so.0)
frame #37: clone + 0x6d (0x7f77efeab8fd in /lib64/libc.so.6)

From the error message,
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.3) Incorrect unused parameter detection. The return value of the `forward` function is inspected by the distributed data parallel wrapper to figure out if any of the module's parameters went unused. For unused parameters, DDP would not expect gradients from then. However, if an unused parameter becomes part of the autograd graph at a later point in time (e.g., in a reentrant backward when using `checkpoint`), the gradient will show up unexpectedly. If all parameters in the model participate in the backward pass, you can disable unused parameter detection by passing the keyword argument `find_unused_parameters=False` to `torch.nn.parallel.DistributedDataParallel`.

parameters before the checkpoint call shouldn't be used again. So the loop of loss += p.sum() should loop only over the unused parameters. Try something like:
unused_params = [module.longformer.pooler.dense.weight, module.longformer.pooler.dense.bias]
loss = loss.mean()
for p in unused_params:
    loss += 0.0 * p.sum()

I think I got slightly farther but it's giving me the same error. When I try to print the unused parameters, it doesn't print those two layers anymore. But I get the same error and I guess there must be something unused somewhere else:
/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
Traceback (most recent call last):
  File "ddp.py", line 205, in <module>
    main()
  File "ddp.py", line 57, in main
    train_model(0, train_dataset = Train_dataset)
  File "ddp.py", line 164, in train_model
    loss, logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 528, in forward
    self.reducer.prepare_for_backward([])
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

I am posting my entire train_model() function below as reference:
def train_model(gpu, train_dataset):

    dist.init_process_group(backend='nccl', init_method='env://', world_size=1, rank=0)
    model = LongformerForSequenceClassification.from_pretrained('allenai/longformer-base-4096',
                                                                gradient_checkpointing=True,
                                                                num_labels=4)

    torch.cuda.set_device(gpu)
    model.cuda(gpu)

    batch_size = 1 # CHANGE BATCH SIZE HERE
    epochs = 1 # CHANGE NUM EPOCHS HERE

    optimizer = AdamW(model.parameters(),
                  lr = 2e-5,
                  eps = 1e-8
                )

    model = nn.parallel.DistributedDataParallel(model, find_unused_parameters=False)

    train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset,
                                                                    num_replicas=1, # World size
                                                                    rank=0) # Only one node, so rank=gpu

    train_dataloader = torch.utils.data.DataLoader(dataset=train_dataset,
                                               batch_size=batch_size,
                                               shuffle=False,
                                               num_workers=0,
                                               pin_memory=True,
                                               sampler=train_sampler)

    total_steps = len(train_dataloader) * epochs

    scheduler = get_linear_schedule_with_warmup(optimizer,
                                            num_warmup_steps = 0,
                                            num_training_steps = total_steps)

    for epoch_i in range(epochs):
        total_train_loss = 0

        model.train()

        for step, batch in enumerate(train_dataloader):
            b_input_ids = batch[0].cuda(gpu)
            b_input_mask = batch[1].cuda(gpu)
            b_labels = batch[2].cuda(gpu)

            model.zero_grad()

            loss, logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

            unused_params = ['module.longformer.pooler.dense.weight', 'module.longformer.pooler.dense.bias']
            loss = loss.mean()
            for n, p in model.named_parameters():
                if n in unused_params:
                    loss += 0.0 * p.sum()

            total_train_loss += loss.item()

            loss.backward()

            # check parameters with no grad
            for n, p in model.named_parameters():
                if p.grad is None and p.requires_grad is True:
                    print('No forward parameters:', n, p.shape)

            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            optimizer.step()

            scheduler.step()

Thanks @ibeltagy -- and for your minimal example. I also found that for HuggingFace models were I add grad checkpointing -- things work ok if DPP is run with find_unused_parameters=False (but fail with thatExpected to mark a variable ready only once...  error otherwise).
I'm still a bit shocked that DDP is much slower than DP and with half the per-GPU batch size [when testing 1 GPU only]. Maybe that will improve in the future, but for now happy that grad checkpoint and DDP do work at all, and for multi-GPU.
According to your test the model I was working with (HuggingFace T5) does not have any unused params that expect gradient. But somehow the find_unused_parameters check still needs to be turned off, and then things train smoothly.

Hi, I am trying to use gradient checkpointing in combination with DistributedDataParallel (4 GPUs, 1 node) and weight sharing between different parts of my model.
My model comprises of a few parts:

Encoder (Roberta checkpoint)
Pooler
Decoder (Adapted Roberta checkpoint)
Language model head

Ideally, I would like to share the weights between the Encoder and Decoder, but I think this does not work in combination with gradient checkpointing and DDP. But I am not entirely sure, as the errors are not always very illustrative.

When I use 1 GPU and no DDP and weight sharing, the gradient checkpointing works fine and increases my batch size from 3 to 20!
When I use 4 GPUs and *DDP and no weight sharing, the gradient checkpointing works (with find_unused_parameters=False and returning only the loss from forward) and that allows me to use a batch size of 8 (so effectively 32)
When I try to 2 GPUs and DDP and weight sharing, the following error is thrown:

RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons:
1) Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes
2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.
3) Incorrect unused parameter detection. The return value of the forward function is inspected by the distributed data parallel wrapper to figure out if any of the module's parameters went unused. For unused parameters, DDP would not expect gradients from then. However, if an unused parameter becomes part of the autograd graph at a later point in time (e.g., in a reentrant backward when using checkpoint), the gradient will show up unexpectedly. If all parameters in the model participate in the backward pass, you can disable unused parameter detection by passing the keyword argument find_unused_parameters=False to torch.nn.parallel.DistributedDataParallel.

Does weight sharing fall in category 2 of this Error message?
It would be great to leverage the benefits of DDP, multi-GPU, gradient checkpointing and weight sharing, but I am just not sure whether that is possible.

Hi, I am trying to use gradient checkpointing in combination with DistributedDataParallel (4 GPUs, 1 node) and weight sharing between different parts of my model.
My model comprises of a few parts:
1. Encoder (Roberta checkpoint)

2. Pooler

3. Decoder (Adapted Roberta checkpoint)

4. Language model head

Altogether they form a nn.module. But gradient checkpointing can be activated for the Encoder and Decoder separately.
Ideally, I would like to share the weights between the Encoder and Decoder, but I think this does not work in combination with gradient checkpointing and DDP. But I am not entirely sure, as the errors are not always very illustrative.
* When I use **1 GPU** and **no DDP** and **weight sharing**, the gradient checkpointing works fine and increases my batch size from 3 to 20!

* When I use **4 GPUs** and **DDP** and **no weight sharing**, the gradient checkpointing works (with `find_unused_parameters=False` and returning only the `loss` from `forward`) and that allows me to use a batch size of 8 (so effectively 32)

* When I try to **2 GPUs** and **DDP** and **weight sharing**, the following error is thrown:

RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons:
1) Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes
2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases yet.
3) Incorrect unused parameter detection. The return value of the forward function is inspected by the distributed data parallel wrapper to figure out if any of the module's parameters went unused. For unused parameters, DDP would not expect gradients from then. However, if an unused parameter becomes part of the autograd graph at a later point in time (e.g., in a reentrant backward when using checkpoint), the gradient will show up unexpectedly. If all parameters in the model participate in the backward pass, you can disable unused parameter detection by passing the keyword argument find_unused_parameters=False to torch.nn.parallel.DistributedDataParallel.

MY QUESTION IS: Is this expected? Does weight sharing fall in category 2 of this Error message?
It would be great to leverage the benefits of DDP, multi-GPU, gradient checkpointing and weight sharing, but I am just not sure whether that is possible.

Hi Claartje,
I met the same problem. Have you solved it?

Hi，@ @moscow25 I also face this problem when i use t5 model in DDP mode and set gradient_checkpointing=True. I find it will consume very more GPU memory than use single GPU when set gradient_checkpointing when batch_per_device is equal. Have you met this problem and any comment for this? Thank you!

I've basically exhausted all possible ways but still could not find a solution for using HuggingFace T5 with gradient checkpointing plus DDP. I am pretty sure there is no unused parameters in my model, and I also tried to do _set_static_graph(), but looks like it does not work for T5. Very frustrated..Sigh..


Top Navigation - Stride Logo disappears when the theme is Auto

From: Joreyk

This bug still exists in the blog page, but not the homepage.

Oh yeah, sorry, refreshing worked.

Caching has advantages but also disadvantages, so these specific bugs are rather caught before the new website is cached by everyone :) At least we know that refresh was working! Thank you @NicusorN5  for testing it for us!


rumtime error in colab

can you run the labeled dataset i have created bro. has i am able not able run the time .
it would be helpful for me as well as useful for you in experimentation
[https://drive.google.com/drive/folders/1Sjg9Pv5wblkNAdv2sj7DIwW4XJOrvrRW?usp=sharing]
if possible other wise reply with "no"

No. Sorry, I don't have time ☹. Hope you can get it figured out! You can also pay $10USD (not sure what it is in other countries) for a month of Google Colab Pro, which increases the session runtime.


chore: bump Cargo version

coverage: 95.796%. remained the same
when pulling 8f873070851b12a2580d2b70383759da6ff5c084 on 1.3.0-alpha
into a3ac9e1011a6f26f44fea8091eab9effca5bbb9b on master.


Replace onOnline/onOffline by onSessionConnected/onSessionDisconnected/onSessionClosed

onSessionDisconnected: indicates the disconnection of the session and the internal turms driver may reconnect the server.
onSessionClosed: indicates the close of the session and the internal turms driver will definitely not reconnect the server.
We don't use the names "onOnline" and "onOffline" anymore because they are confusing for developers to know what happened exactly.

Done in #447


restructure backend model calls

combine and create new classes so that different models can be used interchangeably and reduce redundancies

[x] combine classes for embedding-model and umap-model call
[x] create unique time-stamp directories where embeddings are saved as .pickle files
[x] ensure checking if embeddings already exist before recomputing
[x] align imports so that only module's are imported when needed

classes exist now the calling to get the umap embeddings just has to be implemented


[ie/Kenh14] Merge "master"; simplify the code; add more test variants

Hello! https://github.com/yt-dlp/yt-dlp/pull/3996

merged in https://github.com/yt-dlp/yt-dlp/pull/3996/files/88f6e15a807e8ed775b1643bce7fb093b3508cb8..e31f238dca21a0f3990eea3a8d089dfc5d9a86f6

Nice.


Fixed 'Show Franz' button in tray

Description
When click 'Show Franz' button in the tray, Franz doesn't opening and i've fixed that.
Motivation and Context
#131
How Has This Been Tested?
That changes doesn't affect anything in app.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

Good point 🤘 Removed unnecessary parameter from src/index.js and deleted constructor.


Subscriber recieve hangs infinitely

What pyzmq version?
infinitely
What libzmq version?
no
Python version (and how it was installed)
3.10.5
OS
Windows 11
What happened?
A bug happened!
Code to reproduce bug
That's the server code:

# -*- coding: utf-8 -*-

#from modules.telegram_client import telegram_client
#from settings import Settings

# tg_api_cred = Settings.get_telegram_api_credentials()
# tg_client = telegram_client.TelegramClientCustom('session_name', tg_api_cred['api_id'], tg_api_cred['api_hash'],
#                                                  raise_last_call_error=True)
#
# tg_client.start(Settings.get_telegram_phone_number())

import zmq
import threading
import json
import os
import time

# Connect the publisher socket
context = zmq.Context()
publisher = context.socket(zmq.PUB)
publisher.bind("tcp://<IP_ADDRESS>:5556")

# Connect the router socket for receiving acknowledgements
router = context.socket(zmq.ROUTER)
router.bind("tcp://<IP_ADDRESS>:5557")

# Save message info to a dictionary (a stand-in for your database)
messages = {}

# Send messages
consumers = ['consumer1',  ]

for i in range(10):
    message_id = str(i)
    file_path = f'./content/{i}'

    # Save message info to the database
    messages[message_id] = {
        'file_path': file_path,
        'consumers': consumers.copy(),  # copy the list because we're going to modify it
        'processed_by': [],
    }

    # Send the message to all consumers
    publisher.send_json({
        'message_id': message_id,
        'text': f'This is message {i}',
        'media_path': file_path,
    })

# Cleanup process
def cleanup():
    while True:
        for message_id, message in messages.items():
            print(message.items())
            print(message['processed_by'])
            if set(message['consumers']) == set(message['processed_by']):
                print(f"Deleting file {message['file_path']}")
                # os.remove(message['file_path'])  # uncomment this to actually delete the file
                del messages[message_id]

        time.sleep(5)  # pause between cleanup runs

cleanup_thread = threading.Thread(target=cleanup, daemon=True)
cleanup_thread.start()

# Receive acknowledgements
while True:
    # Wait for next request from client
    message = router.recv_json()
    print(f"Received request: {message}")

    # Process the acknowledgement
    if message['message_id'] in messages:
        messages[message['message_id']]['processed_by'].append(message['consumer'])

    time.sleep(5)

That's the client code:
import zmq
import time

# Prepare context and sockets
context = zmq.Context()
consumer_id = 'consumer1'  # change this for each consumer

# Connect the subscriber socket
subscriber = context.socket(zmq.SUB)
subscriber.connect("tcp://<IP_ADDRESS>:5556")
subscriber.setsockopt_string(zmq.SUBSCRIBE, '')

# Connect the dealer socket for sending acknowledgements
dealer = context.socket(zmq.DEALER)
dealer.identity = consumer_id.encode()
dealer.connect("tcp://<IP_ADDRESS>:5557")

# Process messages
while True:
    message = subscriber.recv_json()
    print(f"Received message: {message}")
    # Send acknowledgement
    dealer.send_json({
        'message_id': message['message_id'],
        'consumer': consumer_id,
    })

    time.sleep(2)  # pause between processing messages

Traceback, if applicable
Consumer just hangs and does nothing, reason unknown. This example was generated by chatGPT v4 and looks fine, but for some reason it won't work

More info
No response

Here's the explanation https://stackoverflow.com/questions/76395082/pyzmq-client-hangs-forever-on-receiving-data/


Authorization header dropped accidentally because wrong port change detection logic

Summary.
Expected Result
We have a library code using requests, when we get a URL with auth, it will be redirected to cas authentication url which is exactly same schema, same hostname, but for the port, our original request explicitly gave 443, but the location returned in 302 response is without ":443" (means default 443 https), in such case we expect the redirected request should not strip the authorization header.
Actual Result
The authorization header is stripped and causing eventually 401 error.
Reproduction Steps
import requests
session = requests.Session()
session.auth = ('admin', 'password1')
response = session.get('https://example.com:443/foo')

System Information
$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.5"
  },
  "platform": {
    "release": "4.15.0-36-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.20.0"
  },
  "system_ssl": {
    "version": "1010008f"
  },
  "urllib3": {
    "version": "1.24"
  },
  "using_pyopenssl": false
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Hi @quickwind, thanks for bringing this to our attention. I've quickly tossed together #4851 to address the problem. I'm not ecstatic about the solution, but it should resolve the issues. I'll try to give the issue some more thought in the morning and we'll work towards getting this resolved in the next bugfix release.

No problem @quickwind, we'll likely wait another week to see if anything else crops up and then I'll look at getting a hotfix release out. Going to close now that #4851 is merged.


Could not compile template - Config option template not working anymore

Environment

Operating System: Linux
Node Version:     v22.6.0
Nuxt Version:     3.14.0
CLI Version:      3.15.0
Nitro Version:    2.10.2
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      -
Runtime Modules:  -
Build Modules:    -

Reproduction
Reproduction of error with nuxt 3 https://stackblitz.com/edit/github-uuytf8-ff5rwk?file=comp%2Fc2.vue,comp%2Fc1.vue,app.vue,nuxt.config.ts,comp.js.template

Example working correctly with nuxt 2 - https://stackblitz.com/edit/nuxt-starter-9qz2v9?file=nuxt.config.js,comp.js.template,pages%2Findex.vue

Describe the bug
Nuxt 3 webpack alias is not working with templates option

Additional context

Full static build

Both nuxt 2 and 3 have components auto import disabled with dirs: []

Problem ocurrs with webpack and vite

I use templates option to import all files inside a folder easily: import components from '~comp'; imports both c1 and c2.

i found the solution for this, it's a change in the behavior of the lodash/template probably.
In my template file i had a literal interpolation with local variable file
  const file = key.replace('./', '')
  components[name] = () => import('<%= options.importPath %>/' + file)

Just had to change to concat with + to fix
  const file = key.replace('./', '')
  components[name] = () => import('<%= options.importPath %>/' + file)

Reopening because i fixed the problem in my code but the error still exists

CustomSwiper.vue. Consider restricting the plugin to particular files via options.include
[plugin inject] shared/components/main/MainHeader.vue: rollup-plugin-inject: failed to parse /var/www/almatytourism.crocos.kz/shared/components/main/MainHeader.vue. Consider restricting the plugin to particular files via options.include
[plugin inject] shared/components/main/MainFooter.vue: rollup-plugin-inject: failed to parse /var/www/almatytourism.crocos.kz/shared/components/main/MainFooter.vue. Consider restricting the plugin to particular files via options.include
[nitro]  ERROR  RollupError: shared/components/ui/UiButton.vue (1:0): Expression expected (Note that you need plugins to import files that are not JavaScript)
1:
I get same error when trrying to build for production.

I've updated your code here to point to the correct path + write the file to disk - and indeed the template is being written correctly.
However, require.context isn't valid and is throwing an error, but that's an issue with your code or webpack. The options are working as expected.
Note that I would recommend instead using addTemplate within a module, as they are really meant for virtual files within the .nuxt/ directory.

Thank you dainelroe for taking the time to look into my issue. I managed to fix my code adding defineAsyncComponent to the template
components[name] = defineAsyncComponent(() => import('<%= options.importPath %>/' + file))

and a combination of addTemplate and extendWebpackConfig
import { defineNuxtModule, addTemplate, extendWebpackConfig } from '@nuxt/kit'
import { readFileSync } from 'fs'

export default defineNuxtModule({
  setup () {
    const data = readFileSync('./lib/buildTemplates/pageComponents.includes.js.template', { encoding: 'utf8', flag: 'r' })
    let parsedContent = data.replaceAll('<%= options.importPath %>', '~/components/common')
    parsedContent = parsedContent.replaceAll('<%= options.recursive %>', 'false')

    addTemplate({
      dst: './components/common/.dist/index.js',
      filename: 'index.mjs',
      getContents: () => parsedContent,
      write: true
    })

    extendWebpackConfig((config) => {
      config.resolve.alias['~includes'] = '~/components/common/.dist'
    })
  }
})


Optimize FileSystem Dock filtering

Sames as #61371, but for FileSystem dock.
Part of #93829

Needs rebase, if still relevant. CC @Hilderin

This changes how FileSystem's tree is managed and it's unrelated to any recent changes. It's still relevant.


[v1.1.7] options.outputPath {Function} is appended to the original URL

Prior to this release, providing an outputPath function would see an argument passed in to it of the current URL of the file. We were using a name like: "[path][name]-[hash].[ext]", and we'd see an argument like "client/fonts/font-icons/icons-123456789.svg" given to the function. The return value of that outputPath function would end up being where the file would be emitted -- the function basically acted like a way to transform the file path. So, for instance, I would use it to just organize some things, e.g.
if (url.startsWith("client/") return url.replace("client/", "assets/");

So, instead of my file being output and served up with a URL of "site.com/client/fonts/font-icons/icons-123456789.svg", it would instead be "site.com/assets/fonts/font-icons/icons-123456789.svg". I was using this to basically help organize the output of some assets for in-house vs. vendor vs. other based on the path prefix I was getting.
With this latest change however (due to the removed "else if" conditional on line 47 of the previous version of the file), the URL gets appended to the result of the output path function (see the change on line 47 of the current version, outputPath += url). Before, in the "else if" that would handle outputPath being present, the url itself was being replaced by the result of the output path (previous version line 51, url = outputPath).
Now, the emitted path ends up being like "assets/fonts/font-icons/icons-123456789.svgclient/fonts/font-icons/icons-123456789.svg", and the re-written URL ends up trying to serve it up at "site.com/client/fonts/font-icons/icons-123456789.svg".

@michael-ciniawsky can you provide failing test? Difficult to understand :smile:

@evilebottnawi, an example.
fonts.scss
@font-face {
    src: url('../fonts/font.woff');
}

webpack.config.js
test: /\.woff$/,
loader: 'file-loader',
options:  {
    name: '[name].[ext]',
    outputPath: 'fonts/',
}

correct bundle.css with 1.1.6
@font-face {
  src: url(/static/./fonts/font.woff);
}

wrong bundle.css with 1.1.7
@font-face {
  src: url(/static/font.woff);
}

@thasmo hm, only if publicPath not set, we should set url as outputPath + url. Seems it is normal, in near future wait patch release.

Same error too, glad to see the patch soon :)

@evilebottnawi @michael-ciniawsky This is fantastic, thank you so much! Sorry if the initial example was a bit unclear. I just checked out 1.1.8 and all is well again!

Hi @michael-ciniawsky @evilebottnawi,
My app have the 1.1.9 version, and I have the same bug than the 1.1.7 version.
My config is :
file: {
      test: /\.(png|jpg|ico|gif|svg|eot|eot|otf|ttf|woff|woff2)$/i,
      loader: 'file-loader',
      options: {
        publicPath: `${process.env.ROOT_URL || ''}/assets/`,
        // output path is relative to the JS bundle folder
        outputPath: 'packed/',
     },
},

Before my assets had the url : servername/assets/packed/123556456461...png
Now I have servername/assets/1231231231215...png

@Themandunord all right, you use bug when output and public is concatenating, use publicPath only when you should change public path

@evilebottnawi
Ok but with this config :
Hi @michael-ciniawsky @evilebottnawi,
My app have the 1.1.9 version, and I have the same bug than the 1.1.7 version.
My config is :
file: {
      test: /\.(png|jpg|ico|gif|svg|eot|eot|otf|ttf|woff|woff2)$/i,
      loader: 'file-loader',
      options: {
        outputPath: 'packed/',
     },
},

My images have  : servername/assets/packed/123556456461...png
But my fonts (from Material-UI) have servername/assets/1231231231215...png
Do you see the problem ? I missed something ?

@Themandunord try to remove outputPath also and check you file-loader version in node_modules

@evilebottnawi Thank ! It's work :)

@evilebottnawi Hum... I don't understand very well finally.
I have :
file: {
      test: /\.(png|jpg|ico|gif|svg|eot|eot|otf|ttf|woff|woff2)$/i,
      loader: 'file-loader',
      options: {
        publicPath: `${process.env.ROOT_URL || ''}/assets/`,
      },
    },

And all my assets are put in the /assets folder like that :

Event if i put :
publicPath: \`${process.env.ROOT_URL || ''}/assets/packed`

What I don't understand ?
I don't want the file in the assets folder, because I can't add them to the .gitignore...


Congratulations in making Cairo Shell a very positive and enjoyable experience.

I really like Shells/ Desktop Environments and softwares that increase productivity.
Quick and easy hotkeys, Fast reaching menus, etc.
First of all thank you for working on this project, i guess your personnel isn't the biggest, because the project has advanced slowly , little by little. Congratulations on your hard work.
1- If i type the letter T while being on the desktop it should highlight a file/folder that starts with the letter T .
So i can just press enter and open it (i use only my keyboard to navigate my windows explorer.exe for example, i do it fast)
2- If i click on file in desktop it should highlight showing me its full name ( That way i dont have to hover on top of it and wait 0,1 or 0,2 secs for it to pop (fast way of seeing and reaching files/folders)
2.1 Capability of Cycling/Passing through Icons/Files/Folders in Desktop with the Arrow keys just like in explorer.exe
3- The scroll bar in the bottom for the Shortcuts in Desktop scrolls slowly in Windows 7 (not fast enough)
4- I really liked the idea of Shift + Win + D for dynamic desktop  (but its to many hotkeys, windows does this with Win + D , i haven't tried if i can only hook 2 combinations of hotkeys.
(Shift + Win + D are 3 key presses it takes longer than Win + D, thats not convenient for my fingers and muscle reactions, if its possible to only hook 2 hotkeys, then this is perfect dont touch it)
5- If i drag my mouse in the Desktop it should be able to create boxes.
6- if i drag an icon in Desktop i should be able to place it wherever the F' i want. (thats more convenient and fast for my eyes to see and reach) Not possible in Cairo Shell, my custom organized icons get messed with your Alphabetic Order.
(i dont like that, because maybe i have a file that is related to another file but they're not named with the same 1st letter, and they'll be to appart of each other)
7- Ability to integrate the shell as default without breaking explorer.exe (i think this will be the hardest , because i see in all updates since 2018-2019 , this hasn't changed).
8- Ability to keep icon position after installing Cairo Shell.
9- Ability to press Win key and open the Cairo Shell menu instead of Shift + Win + Z
10- Cairo shell, Cairo Desktop Environment....... i'd really love to see this as file with 60-220mb's size.
I was looking for something that would ressemble The KDE and XFCE experiences,  With Themes that will apply on Control Panel and Windows settings... But i guess that will be too hard.
Just like KDE - Desktop Environment for example, wiith many many interfaces to setup and config things, even certain Linux OS's will give compatibility errors and nasty things can happend.
SO i understnad this project is very hard to accomplish.
If you accomplish all of these fixes in future updates, or at least some of em, i'm sold, i'll be using Cairo Shell Desktop Environment as my main Environment Experience, and quit Windows explorer.exe.
I would even donate and recommend your software, and install it on other machines.
If you make this project more customizeable, user friendly and less Static, i guarantee more people will like it and enjoy it.
Once the user sees that he cannot drag icons, or make boxes he notices a feeling of things bein too static and in the same place. Bars should be able to move , horz , vert , etc, (moving the bars is not fundamental but still useful).
But i've shown the software to people and they draw back from it because of these things too. They feel icons can't move and are misplaced.
Good luck with your endeavours <3

Also why make a back arrow and go forward arrow.
When you can just alt + left arrow  to go to previous folder
and alt + right arrow to go to next folder.
If i need to go home just win + d
It's nice that that we can disable that bar , but its not really useful.


Stricter type checking of messages

Have a validation function for each message type. Validate each message before processing it. This would help to move all the type checking out of the main codebase (where it's sort of an afterthought now) and make it more rigorous.
Should help to prevent future security issues like https://github.com/feross/bittorrent-dht/issues/87
cc @mafintosh

I think we can close this now that we're using safe-buffer.


feat: alternativepath support

Thanks for this project!
I started something similar years ago but never saw it to completion https://github.com/jeswr/shacl-query; might be able to extract some bits of it though to cover more types of shapes.
This PR hackily adds support for alternativePaths which is something that I needed support for on something I was doing today.
I'm happy to clean this up if you can guide me on how to correctly extract the values pattern in the shapeToPatterns file.

Hey. Thank you so much for this. The internal API is not yet finalised. I have recently been thinking about extensibility, such as that constraints, including custom shape properties can be plugged in. Just FYI
Let me have a look at your PR now. It is a draft because you're still working on it or simply so that I don't merge yet?


count_normalization is only correct for batch_norm. wrong flops count for layernorm

https://github.com/Lyken17/pytorch-OpCounter/blob/43c064afb71383501e41eaef9e8c8407265cf77f/thop/profile.py#L32
The same count_normalization function is used for every norm-esque module but batchnorms store an estimate mean and stdev, while layernorms calculate them at inference time. Shouldn't layernorms account for the cost of evaluating mean and stdev? The difference is pretty significant:
The mean is n flops, stdev is 2n more flops? and thats before the rest of the norm module which is another 2n.
Is there a reason layernorms should be estimateable as only 2n flops by re-using batchnorm's estimate?

I think you are right.
BTW, I think the MACs of BN (eval, no affine) should be n, not 2n in codes.
https://github.com/Lyken17/pytorch-OpCounter/blob/43c064afb71383501e41eaef9e8c8407265cf77f/thop/vision/basic_hooks.py#L60-L69
https://github.com/Lyken17/pytorch-OpCounter/blob/43c064afb71383501e41eaef9e8c8407265cf77f/thop/vision/calc_func.py#L43-L45


Check XDG/FHS locations

As @blackhole89 has already written in #36, cLaTeXMath is not checking locations for it's resources. The best way to do this, is to have a hidden file (.clatexmath-res_root) inside the resource directory. Then check the list of XDG/FHS complainant paths to see if one of them contains the file. I've written a small sample how this could be done - tho it requires c++17 and cLaTeXMath does not compile with std=c++17.
https://gist.github.com/sp1ritCS/9c81ae26a283ec5102c59a77c7fb5c3c

I'm not familiar with XDG/FHS, but this project is completely compatible with C++17, and I'm glad to upgrade to C++17 (some legacy code need to be removed, and I'm now focusing on introducing open math and needs some C++17 features), any PR is appreciated :smiley:

if I change cpp_std to c++17 it throws a lot of error: ISO C++17 does not allow dynamic exception specifications
See build log with warning_level 0: https://gist.github.com/sp1ritCS/57675e4564c84f2b3e4cde82ac3bad4d

I see... The "dynamic exception specifications" should be removed if introduce c++17. I'll make a PR to do this.

I make a PR #39 that remove dynamic exception specifications and upgrade CMake build to c++17, the Meson build could be updated.


[RESOLVED] High ICMP ping response time

Problem started at 02:22:14 on 2023.05.25
Problem name: High ICMP ping response time
Host: ZO-FG
Severity: Warning
Operational data: Value: 193ms
Original problem ID: 22399
Event details in Zabbix: http://<IP_ADDRESS>/tr_events.php?triggerid=23125&eventid=22399

Problem has been resolved in 2m 0s at 02:24:14 on 2023.05.25
Problem name: High ICMP ping response time
Host: ZO-FG
Severity: Warning
Original problem ID: 22399


allow editing

Add a button for editing.  When the app starts, the button should say "Start Editing".  When clicked, the comments text box becomes unavailable, the hex text box becomes editable, and the button should say "Stop Editing".  When clicked again, the hex text will need to be parsed to repopulate the comments text box and the binary MIDI data.  If successful, the file state is set to Edited, the hex text box becomes read-only, the comments text box becomes available, and the button should say "Start Editing".  Otherwise, an error message informs the user of the parsing error.

Resolved with commit c5f72663bfe918133513aad82ca54af69ea95067.


Closing due to no details provided


Feature Request: Delete Boards, including tasks/notes

Is your feature request related to a problem? Please describe.
Suppose I have a Board named, "@Test"
and inside that board, I have a few tasks/notes,
i.e.
@Test
    2. [ ] task 1
    3.  *  note 1
    6. [ ] task 2 (though do note that this is non-sequential)

to delete this, from my understanding, I'd need to run the command
tb -d 2 3 6
While this is fine for most boards, I just went to delete a board that had 20+ items, and as you can imagine, it was a little bit tedious making sure I didn't delete the wrong numbered item.
Describe the solution you'd like
Why not instead of having to type
tb -d 2 3 6
I just have to type
tb -d @Test
And remove all the items in that board (including the respective board)
This is a really cool little tool, aside from this small inconvenience, I'm really enjoying just how usable it is!

I would like to take this one, if there's not a problem :)

I would love to make a PR for that soon

@rjoydip, @raalcaau sorry folks. don't check updates in this issue while making a feature.

@alepop No problem at all. Thanks for the PR

How do i delete board. Is it not possible?


Praj/enable internal es

Description
This is not ready for deploy for production.
This change will let the user manually enable elastic search.
opscode_solr4['external'] = true
opscode_solr4['external_url'] = 'http://localhost:9200'
elasticsearch['enable'] = true
elasticsearch['first_internal_install'] = true
opscode_erchef['search_provider'] = 'elasticsearch'
opscode_erchef['search_queue_mode'] = 'batch'

default['private_chef']['opscode-solr4']['external'] = true
default['private_chef']['opscode-solr4']['external_url'] = 'http://localhost:9200'

default['private_chef']['elasticsearch']['enable'] = true
default['private_chef']['elasticsearch']['first_internal_install'] = true

default['private_chef']['opscode-erchef']['search_provider'] = 'elasticsearch' # solr, elasticsearch
default['private_chef']['opscode-erchef']['search_queue_mode'] = 'batch' # rabbitmq, batch, or inline

https://buildkite.com/chef/chef-chef-server-master-omnibus-adhoc/builds/1066#f9858cfa-1497-465f-a7a6-5c3f2756e824
Check List

[ ] New functionality includes tests
[ ] All buildkite tests pass
[ ] Full omnibus build and tests in buildkite pass
[x] All commits have been signed-off for the Developer Certificate of Origin. See https://github.com/chef/chef/blob/master/CONTRIBUTING.md#developer-certification-of-origin-dco
[x] PR title is a worthy inclusion in the CHANGELOG

Create this as a TODO to not miss out on adding the check: https://github.com/chef/chef-server/issues/1950


Aptly cannot mirror packages.vmware.com/packages/ubuntu repo.

Trying to create mirror:
aptly mirror create -architectures="amd64" -force-components=true "https://packages.vmware.com/packages/ubuntu trusty main" https://packages.vmware.com/packages/ubuntu trusty main
Downloading https://packages.vmware.com/packages/ubuntu/dists/trusty/InRelease...
Downloading https://packages.vmware.com/packages/ubuntu/dists/trusty/Release...
Downloading https://packages.vmware.com/packages/ubuntu/dists/trusty/Release.gpg...
gpgv: Signature made Wed Jun 11 14:18:01 2014 UTC using RSA key ID 66FD4949
gpgv: Good signature from "VMware, Inc. -- Linux Packaging Key --<EMAIL_ADDRESS>ERROR: unable to fetch mirror: architecture amd64 not available in repo [https://packages.vmware.com/packages/ubuntu trusty main]: https://packages.vmware.com/packages/ubuntu/ trusty

My ubuntu trusty vm using this repo without a problem.

seems you did not read the documentation: https://www.aptly.info/doc/aptly/mirror/create/
I was able to go further with:
Usage: aptly mirror create <name> <archive url> <distribution> [<component1> ...]
# it should be
aptly mirror create -architectures="amd64" -force-components=true vmware "https://packages.vmware.com/packages/ubuntu" trusty main
gpgv: Good signature from "VMware, Inc. -- Linux Packaging Key --<EMAIL_ADDRESS>ERROR: unable to fetch mirror: architecture amd64 not available in repo [vmware]: https://packages.vmware.com/packages/ubuntu/ trusty

Finally I checked the vmware repository and it does not fit usual repository directory structure, there is no pool to store the packages for example.

Your command line are basically same as mine :) In the end you got same error as mine.
Nothing in docs forbids to call my mirror repository "https://packages.vmware.com/packages/ubuntu trusty main"
I know that vmware repo is not usual but Ubuntu and apt-mirror working with this repository just fine. I'am expecting from aptly to work with this kind of repositories.

It seems that aptly has not being implemented to support older repositories directory schemes...
https://wiki.debian.org/RepositoryFormat
http://blog.packagecloud.io/eng/2015/08/04/apt-repository-internals/
https://www.debian.org/doc/manuals/repository-howto/repository-howto
Even in the older "obsolete" documentation they talk about pools to manage packages. See "Creating Pools".
So to my mind you should ask vmware to refresh their site in order to comply to a proper apt repository format :)
Regards


feat: add dummy_infrastructure package

Related Issue
https://github.com/autowarefoundation/autoware.universe/issues/1
Related Link
This PR depends on

https://github.com/autowarefoundation/autoware.universe/pull/4

Description
Add dummy_infrastructure packages from https://github.com/tier4/AutowareArchitectureProposal.iv
How to review
Check the following item

Readme
Launch file

https://github.com/autowarefoundation/autoware.universe/issues/92


NEW [WIP] Add localization support with jekyll-simple-i18n

Supersedes #1458. This brings changes from #1503 into the main repo for development.

@djoate:
Preview of index.html translation: https://deploy-preview-1509--privacytools-io.netlify.com/es/
English site: https://deploy-preview-1509--privacytools-io.netlify.com
Sample of partially translated pages:

https://deploy-preview-1509--privacytools-io.netlify.com/es/
https://deploy-preview-1509--privacytools-io.netlify.com/es/providers/ (which you can get to by pressing the green Aprenda Más button under Proveedores on the homepage)
https://deploy-preview-1509--privacytools-io.netlify.com/es/providers/cloud-storage/ (which you can get to by pressing the Servidor Cloud link in /es/providers/)

Every other page has not been translated (pages without translate: true in front matter are not generated)
This is not meant to be a full-fledged PR but rather a proof of concept for a better solution for localizing the site. If this is an acceptable solution, please make this into a branch in this repository so that we can start localizing using this plugin.
With the jekyll-simple-i18n plugin (MIT licensed), this makes translating the site easier to manage. You should visit the plugin's GitHub repository and read up on it, but here are the features of this plugin from their README:

No external dependencies. Plugins utilize existing Jekyll features.
Source strings and page titles can be placed directly in templates for seamless editing and readability.
A ready-to-translate YAML file that includes all of the canonical source strings is generated every time the site is built.
It's easy to add new languages. Just create a single file that contains the translated strings. Everything else happens automatically.
Custom front matter is added to translated pages that can be used within your Liquid templates.
Optional Transifex integration.
Built-in support for hreflang tags.

It's based on Transifex, but it can be used with a different service such as Weblate. I've made some modifications to the plugin (e.g. renaming transifex to weblate and handling null source text).
This PR includes an example of index.html and a part of card.html (The "Learn More" button text) being translated. The plugin did not seem to work with the github-pages gem, so github-pages was switched with jekyll gem (which is what the current i18n branch does anyway). You can go to https://deploy-preview-1509--privacytools-io.netlify.com/es/ (or build locally) to see the following:

https://deploy-preview-1509--privacytools-io.netlify.com gives the original English site.
Here is a snippet of the source code for index.html:

 {% t Sponsors%}

  {% t New!%} {% t Showcase your brand as a sponsor of PrivacyTools here and support our mission of creating a world free of mass surveillance!%} {% t Learn more...%}

A snippet of resources.html:

 {% t Prefer the classic site? View a single-page layout.%}

  {% capture providers_title %}{% t Providers %}{% endcapture %}
  {% capture providers_page %}/{% if page.language %}{{ page.language }}/{% endif %}providers/{% endcapture %}
  {% capture providers_description %}{% t Discover privacy-centric online services, including email providers, VPN operators, DNS administrators, and more!%}{% endcapture %}

  {% include card.html color="success"
  title=providers_title
  icon="fas fa-server"
  iconcolor="dark"
  page=providers_page
  description=providers_description
  %}

Instead of using keys and two different files, you just wrap the original text around with {% t ... %} tags, and the plugin will automatically key that string (with its own ID) into weblate-source-file.yml. If you are trying to translate things inside of a card, you have to do the same thing as before with capturing text.
The source YAML is generated on build into the root folder of the repo. This source file can then be copied into _data/languages/ and then renamed into one of the languages in the language map to set up a translation. This seems much easier to maintain compared to cross referencing between two different files.
The plugin will also not create multiple keys for duplicates of the exact same string. For example, {% t Worth Mentioning %} will have one key associated with it, and there will only be one key to translate. All other pages that use {% t Worth Mentioning %} will share the same key (however, I've modified it so that, for instance, {% t Worth mentioning %} and {% t Worth Mentioning! %} would have distinct keys)
We would have to replace local links with something like this in order to get the right pages (and I believe external links can be wrapped in translate tags without a problem):
{% t Learn more...%}

A porition of the source file, weblate-source-file.yml, looks like this:
---
Prefer_the_classic_site?_View_a_singlepage_layout.: |
  Prefer the classic site? View a single-page layout.

Providers: |
  Providers

Discover_privacycentric_online_services_including_email_providers_VPN_operators_DNS_administrator: |
  Discover privacy-centric online services, including email providers, VPN operators, DNS administrators, and more!

Learn_More: |
  Learn More

It's a different format when compared to what is currently in the i18n branch now, i.e. it has the format
string_key_id: |
  This is a source string from the site

rather than
"string_key_id": "This is a source string from the site"

If this format doesn't work with Weblate, we can change the plugin so that it generates the latter format.
A sample translation into Spanish (using deepl.com) can be found in _data/languages/es.yml:
---
Prefer_the_classic_site?_View_a_singlepage_layout.: |
  ¿Prefieres el sitio clásico? Ver un diseño de una sola página.

Providers: |
  Proveedores

Discover_privacycentric_online_services_including_email_providers_VPN_operators_DNS_administrator: |
  Descubra servicios en línea centrados en la privacidad, incluyendo proveedores de correo electrónico, operadores de VPN, administradores de DNS y mucho más!

Learn_More: |
  Aprenda Más

Known issues

I don't know of a way to make the plugin translate the strings of permalinks. This means that pages such as https://privacytools.io/es/donate will have to stay as /es/donate for now. Update: See comment below since this is actually not an issue.
Because the plugin uses the actual string as the translation ID/key, there may be collisions (e.g. "Learn more..." and "Learn more!" will have the same ID). To help remedy this, I've modified the plugin so that the IDs preserve capitalization and can also contain periods, exclamation marks, and question marks. I've also modified the plugin so that the max length for an ID is 100 characters.
I've had to modify the Gemfile to not use github-pages gem, so github-pages was switched with jekyll gem (which is what the current i18n branch does anyway). The jekyll-sitemap plugin also had to be explicitly added in order for the site to compile.
Breadcrumbs: I'm not a Ruby programmer, so I'm not going to try to make a solution for the breadcrumbs.

To reiterate, this is a proof of concept for a better i18n solution. Feel free to add this as a branch if this seems like an acceptable solution.

The PR description should link to https://deploy-preview-1509--privacytools-io.netlify.com instead of https://deploy-preview-1503--privacytools-io.netlify.com

Another issue: 404.html won't generate for a specific language, even if you key 404.html and set translate: true

Keying for parts 1 to 4 have been done to key the entire site. Almost all of the content has been keyed as of part 4. Part 5 is to merge master into the i18n development branch to fix conflicts, and to clean up any loose ends such as breadcrumbs. The final check after part 5 is done should be done on this PR where we have a preview for the entire branch.

The plugin will also not create multiple keys for duplicates of the exact same string. For example, {% t Worth Mentioning %} will have one key associated with it, and there will only be one key to translate. All other pages that use {% t Worth Mentioning %} will share the same key (however, I've modified it so that, for instance, {% t Worth mentioning %} and {% t Worth Mentioning! %} would have distinct keys)

This may not be the case, and is breaking Weblate. I'm seeing this 4 times in weblate-source-file.yml, for example:
httpsplay.google.comstoreappsdetails?idorg.torproject.torbrowser_KEY: |
  https://play.google.com/store/apps/details?id=org.torproject.torbrowser

This branch is working on Weblate, and you can register at https://weblate.nablahost.com/accounts/register/ to test it out.
Also, the site doesn't build if an empty translation YAML file exists, they need at least 1 translated key. That's probably a bug that could be fixed.

You're importing the strings into the regex filter without escaping the characters, hmm...

A proposed solution for this is in PR https://github.com/privacytoolsIO/privacytools.io/pull/1524 to encode punctuation (and string length)

Some things that haven't been keyed (yet):

Breadcrumbs
Terminal commands (such as the ones described under https://deploy-preview-1509--privacytools-io.netlify.com/operating-systems/#os) and Firefox config options. I don't think Linux terminal commands and FF options have translations.
The privacy statement

This branch is working on Weblate at https://weblate.nablahost.com/projects/privacytoolsio/website/, and you can register at https://weblate.nablahost.com/accounts/register/ to test it out.

Weblate seems to be down

@djoate I know, I'm having issues with it not sending emails :(

As a side note, would it be possible to serve Weblate with privacytools.io as the root domain?

Not at this time AFAIK.

And I'm not sure what's going on with Weblate, it might be a known issue (https://github.com/WeblateOrg/weblate/issues/3231) or I might just need to look at it tomorrow with fresh eyes. In either case, it allows for anonymous suggestions and it works otherwise, so I'll get registrations figured out at a later time.

Seems like Weblate thinks that there's a newline at the end of every string

I've noticed that for regenerating the source file, it seems the best thing to do is to make sure Jekyll isn't serving the site when the file is regenerated.
e.g.,

if you try to delete a string while the site is still being served and you save the file, that string won't be removed from the source file until you stop serving the site and reserve/rebuild the site.
if you add a new {% t string %} and save it while the site is being served, the source file will only add that new string to the end of the file rather than nearby other strings (affects the Weblate nearby strings view) until you stop serving the site and reserve/rebuild the site.

I've noticed that for regenerating the source file, it seems the best thing to do is to make sure Jekyll isn't serving the site when the file is regenerated.

I have noticed that as well.

Seems like Weblate thinks that there's a newline at the end of every string

I saw that but didn't get a chance to look further. The issue, I believe, is this space between each key:
About_PrivacyTools_18_KEY: |
  About PrivacyTools

About_the_PrivacyTools_organization_and_contributors_to_the_PrivacyTools_website_communities_and_servicesP_109_KEY: |
...

In YAML, the | means "all text until the next key" I believe, so technically the source file also has newlines at the end of each string, it just isn't as noticeable in this format, but becomes noticeable when Weblate converts it into a single-line format.
What I don't know is whether or not the \n actually affects anything. I didn't see it change anything on the site itself (and can't imagine when it would make a difference, since newlines are generally ignored in HTML). But if it does affect things removing that space between keys in the source would probably fix it.

The plugin itself also does .strip on all rendered tag text. Regardless, I think leaving the newline for every string may be confusing and that it would be better to try to get rid of it

At a second glance with Weblate, I'd agree:

I've "solved" the email problem temporarily by enabling Sign in with GitHub and GitLab. So feel free to register an account, just don't sign up with email :)

@JonahAragon I looked it up (https://yaml.org/YAML_for_ruby.html#three_trailing_newlines_in_literals). Apparently | would give a final new line while |- strips all newlines

Hi.
Would it be possible to reopen discussions around translations?
Could we put it as a main priority for the community to do every changes needed to make translations a thing with Weblate?
This issue struggles since a long time but is, IMHO, one of the most important one.
Actually, most translations of the website are aged and people should not totally rely on it.

@Booteille My thoughts on the state of translations progress is given at https://github.com/privacytools/privacytools.io/pull/1509#issuecomment-682567897


Lists should use just

At the moment a list like

Item 1
Item 2

Generates the HTML <li><p> which puts too much spacing. Just <li> is much better.

This seems to be caused by Base's markdown library (that Weave.jl uses internally) wrapping all list items inside of a paragraph.
julia> using Markdown

julia> c = Markdown.parse("""
       1. Test
       2. Test2
       """);

julia> c.content
1-element Array{Any,1}:
 Markdown.List(Any[Any[Paragraph(Any["Test"])], Any[Paragraph(Any["Test2"])]], 1, false)

We could change it to ignore the outer paragraph wrapping inside lists like Markdown appears to do in println, but doesn't do in display:
julia> println(c)
1. Test
2. Test2

julia> display(c)
    1.   Test

    2.   Test2

Thoughts @mpastell?

The simplest fix would probably be to change the CSS to fix the spacing issue. There is no simple fix for this issue in the current markdown writer.


kube-state-metrics serviceaccount installed to kube-system namespace by default

It appears that in a fresh DigitalOcean Kubernetes cluster the kube-system namespace already contains a kube-state-metrics serviceaccount:
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","imagePullSecrets":[],"kind":"ServiceAccount","metadata":{"annotations":{},"name":"kube-state-metrics","namespace":"kube-system"}}
  creationTimestamp: "2021-04-24T16:33:13Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:secrets:
        .: {}
        k:{"name":"kube-state-metrics-token-766fc"}:
          .: {}
          f:name: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-04-24T16:33:13Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-04-24T16:33:13Z"
  name: kube-state-metrics
  namespace: kube-system
  resourceVersion: "288"
  uid: 1e228525-a73b-4a93-83d2-09c06ca719b8
secrets:
- name: kube-state-metrics-token-766fc

This is even though kube-state-metrics was not installed (and is not running in any namespace).
The cluster is at Kubernetes version 1.20.2-do.0 and was created using the Terraform DigitalOcean provider version 2.7.0.
This trips the kube-state-metrics Helm chart version 2.13.2, which also tries to create this serviceaccount:

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "kube-state-metrics" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "kube-state-metrics"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"

In a cluster created a long time ago with Kubernetes 1.18 and using a different version of the kube-state-metrics Helm chart this was not the case.
Is it possible to not create this serviceaccount automatically?

Not only a service account exists already, but also the clusterrole and clusterrolebinding, which both trip up the Helm chart.

Something is quite fishy. I installed kube-state-metrics using the Helm chart as explained above, with values.yaml:
serviceAccount:
  create: false
rbac:
  create: false

But I get lots and lots of errors (example):
E0427 10:23:14.724801       1 reflector.go:156] pkg/mod/k8s.io/client-go@v0.0.0-20191109102209-3c0d1af94be5/tools/cache/reflector.go:108: Failed to list *v1.DaemonSet: daemonsets.apps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "daemonsets" in API group "apps" at the cluster scope

So somehow the service account DigitalOcean installs by default does not have sufficient permissions to actually run kube-state-metrics?

Using the following Helm chart values worked around the issue for me.
serviceAccount:
  create: false
  name: kube-state-metrics
rbac:
  create: false

Apologies for the late note, we fixed this in 1.21.2-do.0 and 1.20.8-do.0, respectively. We do not delete the prior policy files automatically, however, so affected customers will need to delete/overwrite them on their own.


write builtins for various bash functionality

Basically to start this can just be the entire IPC list that pkgcore has implemented in python:
src_install functionality

[x]  doins
[x]  dodoc
[x]  dohtml
[x]  doinfo
[x]  dodir
[x]  doexe
[x]  dobin
[x]  dosbin
[x]  dolib
[x]  dolib.so
[x]  dolib.a
[x]  doman
[x]  domo
[x]  dosym
[x]  dohard
[x]  keepdir
[x]  dosed
[x]  fowners
[x]  fperms
[x]  doheader
[x]  newbin
[x]  newconfd
[x] newdoc
[x] newenvd
[x] newexe
[x] newheader
[x] newinitd
[x] newins
[x] newlib.a
[x] newlib.so
[x] newman
[x] newsbin

sandbox

[x]  addread
[x]  addwrite
[x]  addpredict
[x]  adddeny

output

[x]  einfo
[x]  einfon
[x]  ewarn
[x]  eqawarn
[x]  eerror
[x]  ebegin
[x]  eend

various

[x]  has_version
[x]  best_version
[x]  unpack
[x]  eapply
[x]  eapply_user
[x]  docompress
[x]  dostrip

The bash side of dostrip and docompress are now done, the actual underlying action support will need to be added later in some threaded trigger design.

Closing now since all commands that can be implemented as builtins at least have stubs now. Of the stubs remaining to implement, the sandbox related ones will probably stay no-ops as a different solution will be developed while the others should be implemented as dependency functionality coalesces.


detect BCrypt use

The following:

[x] BCryptDecrypt
[x] BCryptEncrypt
[x] BCryptHash
[x] BCryptHashData
[x] BCryptGenerateSymmetricKey
[x] BCryptGenerateKeyPair
[ ] BCryptGenRandom

Hello @mike-hunhoff
I will work on that rule.

Hi @johnk3r! The only API we have left to cover from this list is BCryptGenRandom. Unfortunately I have not found a public sample that uses this particular API, but for the time being you can create the rule and place it in the rule nursery. Once a public sample is found (or if you already have one in mind) we can then graduate the rule.
I'll assign this task to you, thank you!

@mike-hunhoff

[ ] BCryptGenRandom

Wonderful @johnk3r!
Let's create a new rule under data-manipulation/prng for the generation of random numbers using BCryptGenRandom.

Let's create a new rule under data-manipulation/prng for the generation of random numbers using BCryptGenRandom.

Let's add a new rule since the random number isn't necessarily tied to encryption.

Let's create a new rule under data-manipulation/prng for the generation of random numbers using BCryptGenRandom.

Let's add a new rule since the random number isn't necessarily tied to encryption.

Let's create a new rule under data-manipulation/prng for the generation of random numbers using BCryptGenRandom.

Let's add a new rule since the random number isn't necessarily tied to encryption.
Edit: we could also add other similar functions to this new rule, e.g. https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptgenrandom

Ok man.

Let's create a new rule under data-manipulation/prng for the generation of random numbers using BCryptGenRandom.

Let's add a new rule since the random number isn't necessarily tied to encryption.
Edit: we could also add other similar functions to this new rule, e.g. https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-cryptgenrandom

Ok man.

Hi,
I need help to define the rule name...

Hi,
I need help to define the rule name...