Help

I took your repository as the basis for my course project, I had some difficulties with reworking it, could you help? Not for free of course!

Can I contact you for convenience?
Facebook: https://www.facebook.com/profile.php?id=100036246586835
Skype: timati0008

I am trying to make it so that the search takes place only in one file and not in the whole folder. I thought simply replacing FolderBrowserDialog with OpenFileDialog would solve the problem, but I get an error that I can't figure out.

Hello!
Sorry I have been unavailable this past weekend.
For the change that you are describing changing the FolderBrowserDialog to an OpenFileDialog is a good first step.
Now I assume that you are working in the version <IP_ADDRESS> which resides inside the WPFWordSearch folder, meaning you have replaced the SearchPage.xaml.cs/folderBrowserDialog.
The folderBrowserDialog is used in SearchPage.xaml.cs/btnBrowse_Click to write the selected path to the txtPath.Text property, this (along with the initialisation of the folderBrowserDialog) needs to be replaced with the corresponding code to put the selected file path into the txtPath.Text property (the selected file is on the FileName property of the OpenFileDialog). I also recommend that you add a file filter to the OpenFileDialog where you pre-filter on Word files (doc, docx, docm, ...).
In SearchPage.xaml.cs/btnSearch_Click the selected path is then passed to SearchPage.xaml.cs/search and along to the constructor in SearchProgress.xaml.cs which in turn is used in SearchProgress.xaml.cs/Window_Loaded to initiate the search for Word files in the path using the WordFileLocator.FindWordFiles method. This method returns a string array of paths of Word files.
Instead of using the WordFileLocator.FindWordFiles method you can instead put the path into a new string array and pass it directly to the GetSearchResults function in SearchProgress/Window_Loaded instead of using the result of WordFileLocator.FindWordFiles.
This would look like this:
await Task.Run(() => this.SearchResults = this.search.GetSearchResults(new string[] {this.path}, this.expression, this.isRegex));
I think this should be sufficient to change the search to use a single selected file instead of searching through a whole directory.
Let me know if this was useful or if you're stuck at some point.

Wow, thanks for answering, I thought you'd ignore me. Yes, everything worked out, I didn't think of changing await Task.Run (() in SearchPage.xaml.cs, in the end everything works.

Great! If you need any further help just reply to this issue! Good luck with your course!

In general, I have little time, I need to remake the program, and if I understand myself, I will not meet the deadlines.
I want to ask you to remake the program, there are, in theory, small changes, I am ready to pay $ 15, I don’t know if this is a lot for the CIS countries, correct if not right.
In general, there is a Word document, it contains links directly in the text in this format: [1] and they must be mentioned in order in the text, [1] text-text-text [2] as an example, and I need to make the program so that it checks whether the numbering of links is in order, if there is an error somewhere, then you need to display where the error is.
Sorry if I'm too arrogant or something like that, if you're busy then tell me about it.

From your description I would assume that the following text:
This is text with a[1] link. There is more text with links[2].
Some links[3] are here and others[4] are there.
There’s effin’ links[6] everywhere[5].
And if the order[7] is so wrong,
or if there’s dupes[7] that don’t belong,
we’ll find them.
And if links[9] are all[10] fucked up then there’s gonna be a lot of results[11].

would result in the listing of:

links[6]
everywhere[5]
order[7]
dupes[7]
links[9]

Please check the branch linkOrderSearch which is where I have implemented this behaviour:

Test Word file: test.docx

I probably didn’t put it that way, now I’ll show you an example of the text, don’t look at the language, the point here is just to check if the numbers in brackets are in order.
When developing any Android [1] application, the developer needs to store the data that his application operates on. You can store them both locally on the device and in the cloud storage. For example, in Firebase Cloud Storage. Some applications combine all these methods of storing data. Most modern Android applications are able to synchronize data, store settings and work without the Internet, using data previously recorded in the local storage [2].
The numbers in brackets are simply a reference to some source of literature. Like this:
`BIBLIOGRAPHIC LIST

Deytel, P. Android for programmers: creating applications [Text] / P. Deytel - SPb .: Peter, 2012. - 576 p.
Kotlyarov, V.P. Basics of software testing [Text] / V.P. Kotlyarov - SPb .: Binom, 2009 .-- 496 p.
Goloshchapov, A.L. GoogleAndroid. Creation of applications for smart phones and tablet PCs [Text] / А.L. Goloshchapov - SPb .: Peter, 2012 .-- 315 p.`
And I just need to check if the numbers in brackets are in order (links)

Okay, I am not sure about the output so for convenience I've re-made the thing from scratch and split it up in a library that does the search and an example console application for how to use the search and check the order, so you can use the search in any way you like (eg. with UI or integrated into another project or as a web service etc.). It's not a lot of code and I've commented everything so it should be very easy to understand and modify.
What it does is, it checks if all the numbers inside brackets are in order, like this:
WordLinkOrder/WordLinkOrder.cs is the class for executing the search for all occurrences of [n] in a Word document. The constructor takes the path, the Run method executes the search and returns an int array of all references in order of appearance in the document.
WordLinkOrderChecker/Program.cs is an example on how to use the WordLinkOrder class and also performs a simple check, for each match it will check if the current match is the previous match + 1 (so if matches were [1] [2] [3]...), if not it would output the reference number and the expected reference number, then proceed with the current match as the previous match. The implication is that in the case of [1] [3] [2] [4] it will mark 3, 2 and 4 as bad references because after 1 it'd expect 2 but got 3, then for 2 it would expect 4 as the previous match was 3, and finally at 4 it would expect 3 as the previous match was 2.
I'm still not sure if this is the matching you'd expect but the code is very simple and you can easily adjust it to your needs:

If you need to adjust the order detection, check WordLinkOrderChecker/Program.cs where the foreach contains all code that checks for the correct order.
If you need to adjust the output of the order detection (eg. don't print which references were in bad order, just print either "There are references in the wrong order!" or "All references are in the correct order!", or print all references and mark bad ones in the output) you can edit the foreach inside WordLinkOrderChecker/Program.cs.
If you need to adjust the matching, check WordLinkOrder/WordLinkOrder.cs which just takes a regular expression to search, there you can change the matching pattern.
If you want to add a proper UI you can create a new WinForms/WPF/ASP/... project and add the WordLinkOrder project as reference, execute the WordLinkOrder as done in WordLinkOrderChecker/Program.cs and then handle the resulting int array as required in the UI.

Example:

It's all checked in to the branch linkOrderSearch.

Yes, everything works, I just fixed a small bug associated with the variable "lastNumber" the last number found in the text was taken there, because of this, for example, when finding [22] and after him [8] then it wrote that [23] was expected, although in fact it was expected in general [2], well, this is nothing I fixed, everything works,
`// Check reference order
int firstNumber = 0;
int lastNumber = 0;
bool hasBadReferences = false;
foreach (int reference in references)
{
if (reference != firstNumber + 1)
{
                Console.WriteLine($"Bad reference order found: Got [{reference}] but expected [{lastNumber + 1}]!");
                hasBadReferences = true;
            }

            firstNumber = reference;
            lastNumber=lastNumber+1;
        }`

it remains to attach the whole thing to the visual interface of your program, which I am doing now.
Thank

I had difficulties with the logic of work, I created the main form on which there is just a file selection button and a text line in which the path and the button to start the entire program are written.

`void btnSearch_Click(object sender, RoutedEventArgs e)
{
string path = txtPath.Text;
        if (String.IsNullOrEmpty(path))
        {
            System.Windows.MessageBox.Show("No search path was provided. Please enter or select a path to search in.", "Missing search path", MessageBoxButton.OK, MessageBoxImage.Error);
            return;
        }
        registryKey.SetValue("Path", path);//Register rewriting

        //passing to path function

        WordLinkOrder.WordLinkOrder wordLinkOrder = null;
        try
        {
            wordLinkOrder = new WordLinkOrder.WordLinkOrder(path);
        }
        catch (FileNotFoundException ex)
        {
            Console.Error.WriteLine($"The file {ex.FileName} could not be found.");
            return;
        }
        //Running a check
        int[] references = wordLinkOrder.Run();

        // Check reference order
        int firstNumber = 0;
        int lastNumber = 0;
        bool hasBadReferences = false;
        foreach (int reference in references)
        {
            if (reference != firstNumber + 1)
            {
                //There should be some other code here
                //Console.WriteLine($"Bad reference order found: Got [{reference}] but expected [{lastNumber + 1}]!");
                hasBadReferences = true;
            }

            firstNumber = reference;
            lastNumber = lastNumber + 1;
        }
    }`

I on btnSearch_Click set sending the file path to the WordLinkOrder function, and also launching wordLinkOrder.Run () and immediately checking links, I understand that this is not correct, but I don’t understand how to do it correctly.
My whole solution looks like this:

Search and Program do nothing there, all the code is commented out and the ResulPage page is not implemented, it is empty.
As I understand it, I need to create a class in which there will be the entire miscalculation of the search for links, and in btnSearch_Click, you just need to run this class, but there is still ResultPage how to implement it is not clear at all.

Very good!
You can take the result of wordLinkOrder.Run() (the int array) and pass it to ResultPage and do the calculation there, then display it. I forgot how the result page ist called, but I will provide an example when I get home in a few hours.

Thank you, I will now try to upload the repository to my account and send it to you to make it easier for you to understand the situation!

https://github.com/Wurdy-gear/Link-Checker
So I downloaded the repository, only there the file "v16" I don't know where it came from, it's not needed.

only there the file "v16" I don't know where it came from, it's not needed

Those are files created by Visual Studio that are not required in the repository. For new repositories you should always add a ".gitignore" file that excludes unwanted files, for Visual Studio this is a template: https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
Now the ResultsPage is a Page, which usually uses a NavigationWindow for displaying (in the original code it was MainWindow : NavigationWindow which loaded Pages, like the search page or results page). Your MainWindow however is a Window, so I'm not sure if it's possible to open a page on it. You may have to re-create ResultsPage as a Window too so you can call it using new ResultsPage(references).Show() or something like that.
Also the MainWindow needs to be closed/hidden so I think the code would look like this:
int[] references = wordLinkOrder.Run();
this.Hide(); // Hide current MainWindow
new ResultsWindow(references).ShowDialog(); // Show ResultsWindow modally
this.Close(); // Once ResultsWindow is closed also close this window to exit the app
// alternatively (comment this.Close() and uncomment the next line for this to work):
// this.Show(); // Once ResultsWindow is closed, show the MainWindow again to process another file

Anyway, you can pass the int array to the constructor of ResultsPage (new ResultsPage(references)) and then do the order check that is in MainWindow.xaml.cs line 78-93 in the constructor too. Then display it any way you like on there.

I updated the repository, everything works, it remains to draw a beautiful output, do not tell me where and how to call ResultWinds and what to write in it so that how is the output in your program?

Probably I asked a question that was too incomprehensible, so I will try to specify.
I have an array int [] after working WordLinkOrder, after that i try to process this array in class Check
           //Running a check
           int[] references = wordLinkOrder.Run();
           //System.Windows.Forms.MessageBox.Show("Check запущен и начал проверку");
           // Check reference order
           int firstNumber = 0;
           int lastNumber = 0;
               foreach (int reference in references)
               {
                   if (reference != firstNumber + 1)
                   {
                   //System.Windows.Forms.MessageBox.Show($"Bad reference order found: Got [{reference}] but expected [{lastNumber + 1}]!");
                   //Console.WriteLine($"Bad reference order found: Got [{reference}] but expected [{lastNumber + 1}]!");
                   }
                   firstNumber = reference;
                   lastNumber = lastNumber + 1;
               }
           //There should be a call to a new window with the results
           new ResultWindow(references).ShowDialog(); // Show ResultsWindow modally

And I have the following code in ResultWindow.xaml
<Window x:Class="Practic.ResultWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:local="clr-namespace:Practic"
        mc:Ignorable="d"
        Title="ResultsWindow" Height="450" Width="800">
    <TextBlock Text="{Binding references[]}" TextWrapping="Wrap" Margin="43,32,0,0" />
</Window>

And in ResultWindow.xaml.cs
    public partial class ResultWindow : Window
    {

        public ResultWindow(int [] references)
        {
            InitializeComponent();
        }
    }

But this all does not work, nothing is displayed at all, I am an absolute 0 in visual programming, you can tell me how to make it display an array, and it is necessary that it displays and shows exactly where it was found, just like in your program.

it displays and shows exactly where it was found

What exactly do you mean by "show exactly where it was found"?
The very first version already had this implemented, see the screenshot from this comment:

The int array only contains the numbers of the references, no additional information. If you want more information, like the surrounding text, then this needs to be tracked too so you can't use an int array but have to use a SearchResult class where all the required data is stored, like in the initial commit.

it displays and shows exactly where it was found

What exactly do you mean by "show exactly where it was found"?
The very first version already had this implemented, see the screenshot from this comment:

The int array only contains the numbers of the references, no additional information. If you want more information, like the surrounding text, then this needs to be tracked too so you can't use an int array but have to use a SearchResult class where all the required data is stored, like in the initial commit.
Anyway I recommend that you check out the initial ResultPage on how you can display an array of data (I'm using data bindings, which is fairly easy).

But the repository that you mentioned is the same console project, although in the screenshot, it is not a console project, I don't understand what?

This is implemented in the repository, you did not throw off the code on how to do this:

But the repository that you mentioned is the same console project, although in the screenshot, it is not a console project, I don't understand what?

Yes, it's the commit 80473329f43ab86eb4fb3170c831e50e921ba709.

There are problems with displaying the displayed text.
 the Text that is displayed before and after the found link is programmed here right?
searchResults.Add(new SearchResult()
                        {
                            FilePath = filePath,
                            Match = currentMatch,
                            BeforeMatch = documentText.Substring(Math.Max(match.Index - this.resultRadius, 0), this.resultRadius),
                            AfterMatch = documentText.Substring(match.Index + match.Value.Length, documentText.Length - Math.Min(match.Index + match.Value.Length + this.resultRadius, documentText.Length))
                        });

Can it be configured to render from paragraph to paragraph without the complexity of your original project, i.e. without positionWordStart, positionWordEnd, splitDocument?
More specifically, how can you find the beginning of a paragraph and its end?

And another huge request, can you explain in more detail how this function works?
private string readDocument(string filePath)
        {
            if (Token.IsCancellationRequested)
                return String.Empty;

            this.updateProcess($"Reading file {this.Current + 1}/{this.Total} ({Path.GetFileNameWithoutExtension(filePath)})", this.Current, this.Total);

            Word.Document document = this.application.Documents.Open(FileName: filePath, ReadOnly: true, Visible: false);

            try
            {
                string documentText = String.Join(paragraphSeparator, document.StoryRanges.OfType<Word.Range>().SelectMany(r => r.Paragraphs.OfType<Word.Paragraph>()).Select(p => p.Range.Text)).Replace("\v", paragraphSeparator);

                document.Close(SaveChanges: false);

                this.updateProcess($"Read file {this.Current + 1}/{this.Total} ({Path.GetFileNameWithoutExtension(filePath)})", this.Current, this.Total);

                return documentText;
            }
            catch (Exception)
            {
                return "";
            }
        }

Namely this line
    string documentText = String.Join(paragraphSeparator, document.StoryRanges.OfType<Word.Range>().SelectMany(r => r.Paragraphs.OfType<Word.Paragraph>()).Select(p => p.Range.Text)).Replace("\v", paragraphSeparator);

I want to know how the text that is displayed is created, in as much detail as possible, please.

I need to somehow find a paragraph, as you do when you put a tick in the checkbox paragraph.

How do I find the beginning of a paragraph?
My display is adjusted manually and I want to make it seem like I always have a checkmarked paragraph in the checkbox:

To find the beginning of a paragraph I suggest that you find the previous \n from the location of the match. Same for the end of the paragraph where you'd find the next \n.

And how exactly to search with which function?

Check https://github.com/Longoon12000/WordSearch/blob/80473329f43ab86eb4fb3170c831e50e921ba709/WPFWordSearch/Search/Search.cs#L71
                        searchResults.Add(new SearchResult()
                        {
                            FilePath = filePath,
                            Match = currentMatch,
                            BeforeMatch = documentText.Substring(Math.Max(match.Index - this.resultRadius, 0), this.resultRadius),
                            AfterMatch = documentText.Substring(match.Index + match.Value.Length, documentText.Length - Math.Min(match.Index + match.Value.Length + this.resultRadius, documentText.Length))
                        });

There is a BeforeMatch and AfterMatch in SearchResult which is filled here with the "radius" text. Instead of using radius you have to find newlines.
Example:
documentText.Substring(0, match.Index) gives you all text from beginning to the match, so you can use LastIndexOf to find the position of the newline before the match.Index.
Then you can use IndexOf(..., match.Index) to find the position of the newline after the match.
You can then use Substring on documentText to get the text you want and put it in Before/AfterMatch or replace Before/AfterMatch with a variable where you put in the whole paragraph.

Thank you very much, I did everything, how can I thank you?

No need!
If everything is done you can close this issue.

Thanks a lot again!


why GetLocalGPUInfo() using cuda runtime api, I guess it should use cudart_stub instead.

Base on below commit, cuda Runtime API should wrapped in dynamic load， but I still see  tensorflow/core/grappler/clusters/utils.cc using cuda Runtime API.
commit: 7c5aed39f0148f93cebef850d939558a938b41e7
Remove tensorflow core runtime's explicit dependency on cuda runtime.

Add stub implementation which is wrapped in dynamic loader
Replace the runtime dependency with stub in tf_cuda_library

Adopted csigg@'s method in cl/230858143 to work around the issue where the cuda runtime calls in Eigen::initializeDeviceProp() cannot be wrapped with the macro method.
PiperOrigin-RevId: 232063778

@chsigg any ideas?

@yifeif might have missed them or they might have been introduced afterwards.

@yifeif ，could you please tell me how to fix this issue? I tried use cc_library instead of tf_cuda_library, code path in GOOGLE_CUDA will not execte.

Any suggestion?

@chengleiwang I believe tensorflow/core/grappler/clusters/utils.cc is using cudart_stub.
utils is a tf_cuda_library, which depends on cudart_stub for its implementation.

@yifeif, yes, utils depends on cudart_stub, but actually it doesn't call dynamic cuda runtime api. I have debug this for  several days. You can do a quick check by remove cudaGetDevicrProperties() in cudart_stub.cc/cuda_runtime_10_0.inc and then you can run a test , set a breakpoints at cudaGetDeviceProperties() in utils.cc. You can see it still can execute this Api without using dynamic one.
I see that TensorFlow still using static library during the building step. So I guess this is the root cause for this issue. utils.cc will link to static library's symbol. So may I know why TensorFlow still depends on static library? I tried to remove static link part but I got build errors~ would you pls give me some suggestion? I worked on this issue for long days but no progress ~~

Which platform are you on?

Ubuntu18.04

@yifeif , Are you able to reproduce this issue? Any suggestion?


Incorrect using

In the "Add using directives" (link: https://docs.microsoft.com/en-in/azure/storage/queues/storage-dotnet-how-to-use-queues#add-using-directives), the second and third line should be：
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Queue;

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 2449e264-13a7-ecc0-7400-b6d6bed1b029
Version Independent ID: bcd3da61-efe6-4ef5-1c79-4fa78cdf473d
Content: Get started with Azure Queue storage using .NET - Azure Storage
Content Source: articles/storage/queues/storage-dotnet-how-to-use-queues.md
Service: storage
Sub-service: queues
GitHub Login: @mhopkins-msft
Microsoft Alias: mhopkins

@kuyeduwu , Thanks for the valuable feedback! I have assigned the issue to the content author to investigate further and update the document as appropriate.

@kuyeduwu - Those namespaces have changed with the latest versions of the packages. The article has been updated to reflect these changes. However, there is another section that still referred to an old namespace. So, thank you for prompting me to review the article.
#please-close

@kuyeduwu, we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion


Numbered parameter inside eval is treated as a local variable

ruby -e 'proc { _1 }.call(1)' # OK

ruby -e 'proc { eval "_1" }.call(1)'
(eval):1:in `block in <main>': undefined local variable or method `_1' for main:Object (NameError)

So CRuby treats _1 inside eval even in block as a method call, not local variable. But Prism returns LocalVariableReadNode instead.
Example
It could be illustrated with the following Ruby code snippet:
Prism.parse "_1", scopes: [[], []]

and result:
=>
#<Prism::ParseResult:0x000000010fdf9610
 @comments=[],
 @data_loc=nil,
 @errors=[],
 @magic_comments=[],
 @source=#<Prism::Source:0x000000010fdbef10 @offsets=[0], @source="_1", @start_line=1>,
 @value=
  @ ProgramNode (location: (1,0)-(1,2))
  ├── locals: [:_1]
  └── statements:
      @ StatementsNode (location: (1,0)-(1,2))
      └── body: (length: 1)
          └── @ LocalVariableReadNode (location: (1,0)-(1,2))
              ├── name: :_1
              └── depth: 0,
 @warnings=[]>

As far as I see scopes option affects result and determines whether it will be CallNode or LocalVariableReadNode.
Original code
Original code that triggers this issue is the following MRI test:
# ruby/test_literal.rb

  def test_float
    head = ['', '-', '+']
    chars = ['0', '1', '_', '9', 'f', '.']
    head.each {|h|
      6.times {|len|
        a = [h]
        len.times { a = a.product(chars).map {|x| x.join('') } }
        a.each {|s|
          next if s.empty?
          next if /\.\z/ =~ s
          next if /\A[-+]?\./ =~ s
          next if /\A[-+]?0/ =~ s

          begin
            r1 = Float(s)
          rescue ArgumentError
            r1 = :err
          end
          begin
            r2 = eval(s)
          rescue NameError, SyntaxError
            r2 = :err
          end
          r2 = :err if Range === r2
          assert_equal(r1, r2, "Float(#{s.inspect}) != eval(#{s.inspect})")
        }
      }
    }
    assert_equal(100.0, 0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100e100)
  end

Ahh this is unfortunate. I'll look into this next week.


update mapping for azure-mgmt-privatedns

Docs Build status updates of commit 38dbafb:
:clock10: Full build: the average full build time is 225 min(s) 28 sec(s), this is based on the last 44 build(s) for this repository.

Docs Build status updates of commit 38dbafb:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs-ref-mapping/reference.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


improvement：hope skywalking ui support multi environment data viewing

for users ,this is more convenient,thanks.

What do you mean multi environment data viewing? I am not following.

for example ,you have two environment ,so you will deploy two clusters of collector ,and also elasticsearch. but for ui,  I  want to deploy just one cluster, and there is a drop box ,which can switch environment  to view diffrent data in diffrent elasticsearch clusters.

Oh, I see. This is a complex requirement. I doubt the UI team will be interested in this feature.
ping @hanahmily What do you thing?


Not work with static IP

When I configure my ESP12F with static IP, the firebase didn't not work.
here is my simple code :
#include "FirebaseESP8266.h"
#include <ESP8266WiFi.h>

#define FIREBASE_HOST "xxxxxx.firebaseio.com"
#define FIREBASE_AUTH "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
#define WIFI_SSID "xxxxxx"
#define WIFI_PASSWORD "xxxxxx"

IPAddress staticIP(192, 168, 2, 152);
IPAddress gateway(192, 168, 2, 1);
IPAddress subnet(255, 255, 255, 0);

FirebaseData firebaseData;
String path = "/test";

const int ledPin = 2;

void setup()
{
  pinMode(ledPin, OUTPUT);

  WiFi.config(staticIP, gateway, subnet); // static IP configuration.
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

  while (WiFi.status() != WL_CONNECTED)
  {
    digitalWrite(ledPin, 0);
    delay(200);
    digitalWrite(ledPin, 1);
    delay(200);
  }

  digitalWrite(ledPin, 1); //led off

  Firebase.begin(FIREBASE_HOST, FIREBASE_AUTH);
  Firebase.reconnectWiFi(true);
  Firebase.setString(firebaseData, path, "tost");

}

void loop()
{
    digitalWrite(ledPin, 0);
}

Expected behavior
after connection to router was completed, esp able to send data to firebaase.
Firebase.setString(firebaseData, path, "tost");
IDE and its version:

Arduino
Version 1.8.10

ESP8266 Arduino Core SDK version

Version  2.5.2

It's not relate to the library because it's just software, the IP handles by another stack.
You need to test with ping or http client yourself.

Now I test using : ESP8266WebServer.h :
#include "FirebaseESP8266.h"
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>

ESP8266WebServer server(80);

#define FIREBASE_HOST "xxxxxx.firebaseio.com"
#define FIREBASE_AUTH "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
#define WIFI_SSID "xxxxxx"
#define WIFI_PASSWORD "xxxxxx"

IPAddress staticIP(192, 168, 2, 152);
IPAddress gateway(192, 168, 2, 1);
IPAddress subnet(255, 255, 255, 0);

FirebaseData firebaseData;
String path = "/test";

const int ledPin = 2;

void setup()
{
  pinMode(ledPin, OUTPUT);

  WiFi.config(staticIP, gateway, subnet);
  WiFi.mode(WIFI_STA);
  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);

  while (WiFi.status() != WL_CONNECTED)
  {
    digitalWrite(ledPin, 0);
    delay(200);
    digitalWrite(ledPin, 1);
    delay(200);
  }

  digitalWrite(ledPin, 1); //led off

  Firebase.begin(FIREBASE_HOST, FIREBASE_AUTH);
  Firebase.reconnectWiFi(true);
  Firebase.setString(firebaseData, path, "tost"); // not work

  server.on("/test", blinkLed);
  server.begin();

}

void loop()
{
    digitalWrite(ledPin, 0);
    server.handleClient();
}

void blinkLed() { // it is work
  server.send(200, "text/plain", "ok");
  digitalWrite(ledPin, 0);
  delay(200);
  digitalWrite(ledPin, 1);
  delay(200);
}

I test using http://<IP_ADDRESS>/test , it respones "ok", and led was blingking
But, Firebase.setString(firebaseData, path, "tost"); // it still not work.

You should test for internet not for local server.

Device can't access to the internet due to your network configuration problem.

Thanks mobizt,
I 've tested using ping to internet, The result was false.
Any help related to this problem (static ip to connect to internet) ?

Look like you're using Belkin router.
Try reset or reboot it or change its ip to <IP_ADDRESS> or <IP_ADDRESS>.

Ok mobizt, thank you very much.

If you can change router ip, use ipconfig in Windows cmd to see the changes.

You can change the router IP address and disable DHCP see this on step7.


We need more documentation with flowcharts like this. An excellent resource!

This is an excellent resource!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 510808e8-db5d-6b07-9a04-6edb743a2dfc
Version Independent ID: 20d098aa-ef31-cb26-d677-765cfc9a7164
Content: Decision tree for Azure compute services - Azure Application Architecture Guide
Content Source: docs/guide/technology-choices/compute-decision-tree.md
Service: architecture-center
Sub-service: reference-architecture
GitHub Login: @MikeWasson
Microsoft Alias: pnp

@samsmithnz Glad to hear that this documentation was helpful :)


Fix 1455 and retain invalid frontmatter over failing/overwriting

Turns out people use templating languages and interpolation in frontmatter which makes it invalid unless rendered first. The idea of this PR is to pass-thru frontmatter when attempts to parsing it fails.
Fixes https://github.com/stateful/vscode-runme/issues/1455

Lemme know what you think @adambabik. Making checking for errors optional seemed like the best way forward to solve for the PR's description/linked issue.

@adambabik feedback makes perfect sense 👌. Thank you.


SAK-44157 New feature for users to add a preconfigured layout in Lessons

This adds a new dialog that allows a user to add a layout from a template.
The layout consists of existing Lessons elements but adds them all at once
instead of forcing the user to add each step manually.
Add custom color schemes to layout. The existing color schemes will be
left as "Legacy" colors. Each color scheme has a no borders option.
This also changes edit section to a cog and moves the edit links inside
of the column box.
Improved expand collapse icons
Improve layout of column and section settings dialog
Only show settings relevant to what is being edited. Include new section
for color scheme.
Add show borders checkbox to column settings dialog
Remove the no-borders classes and add a checkbox to toggle borders on
the column dialogue. This matches the new Add Layout options.
Dark themes change the background color all others use a light gray background.
Allow instructors to change color of subpage buttons
Instructors now have the following options:
In the "Edit Subpage" dialog, can choose the color for individual subpage buttons.
In the section settings, force all subpage buttons within to be the same color scheme as the section itself
In the Add Layout dialog, choose to force all subpage buttons within to be the same color or not
In the Add Subpage dialog, choose what color the button should be.

@ern I've rebased and resolved the conflicts.


add support for replacing custom properties (CSS variables) with inline values

In Gmail, and likely in other email clients, a number of CSS 3 properties aren't supported, including custom properties
Currently the custom properties are inlined as-is. I'm not sure if this is a bug or not, as there may be a valid use-cases for retaining the custom properties.
This is likely safer to implement as a configuration option to inline the custom properties.
As an example:
Input:
:root {
  --clr: red;
}

.red {
  color: var(--clr);
}

After inlining:

Currently, css-inline produces the following:

There are a couple cases to handle, with the most top of mind being:

unable to resolve a custom property
handling fallback values
traversing up the cascade for resolution, as custom properties can be scoped within classes, not only in :root

Would love to tackle this, as I'm keen to develop my Rust-fu, but I'm not sure when time constraints would allow, so thought I'd at least open an issue for visibility
EDIT: not so sure about the necessity for this anymore, as providing the fallback in the CSS file renders the correct styles in Gmail anyways. I'm gonna close this, as it seems to be a lot of complexity where CSS has already provided a working solution

Can we reopen this one please? Translating would be super useful since most clients do not support css variables https://www.caniemail.com/search/?s=css+variables
Fallbacks are not enough when you depend on specific colours and fonts being used.

@jarshwah does PostCSS's CSS Variables help, here? They way I realised to use css-inline was to treat it as a tool that processes CSS as-is rather than something that I expect to be able to perform transformations on - it's up to the user to perform any transformations before css-inline evaluates anything

Thanks for pointing that out, but we don't have javascript in our pipeline at all, so unfortunately won't help us here. What we've ended up doing is using template tags (django templates) to render the actual values we want based on some inputs.


Key Error in Notebook '04-gpt2-sentiment-ppo-training.ipynb'

Hi Leandro,
I was running the notebook '04-gpt2-sentiment-ppo-training.ipynb' for the first time, and received a Key Error when running the training loop section. It was in this line:
   rewards = torch.tensor([output[1]["score"] for output in pipe_outputs]).to(device)
I presume it is safe to omit the '[1]'?
   rewards = torch.tensor([output["score"] for output in pipe_outputs]).to(device)
Thanks in advance and best,
Philip

Will have a look!

Hi @lvwerra Any fix on the above error ?
I was running the notebook '04-gpt2-sentiment-ppo-training.ipynb' for the first time, and received a Key Error when running the training loop section. It was in this line:
rewards = torch.tensor([output[1]["score"] for output in pipe_outputs]).to(device)
I presume it is safe to omit the '[1]'?
rewards = torch.tensor([output["score"] for output in pipe_outputs]).to(device)

It should be fixed now!


⚠ Bofrost.fr has degraded performance

In 350655f, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 943 ms

Resolved: Bofrost.fr performance has improved in fe6a225 after 7 minutes.


Window::RecursiveWaitOrCall is performance-heavy

Taking a CPU profile on HoloLens running a release-mode build shows that Window::RecursiveWaitOrCall is fairly heavy on performance-- most likely due to the high number of dispatch calls. It would be beneficial to reimplement this function using a worker thread that sleeps for the given timeout and then dispatches the function.

Now implemented in https://github.com/BabylonJS/BabylonNative/pull/1146.


feat: backport alert improvements from https://github.com/h-enk/doks

This PR depends on changes in https://github.com/BluBracket/docs.blubracket.com/pull/14 and should be reviewed after that is merged.
Source commits:

https://github.com/h-enk/doks/commit/04a5b32ee568cf4b0321f57a779561e99511b1c1
https://github.com/h-enk/doks/commit/1c41ccd5df1564260c908c2b6246d1a988619d8b

Closing in favor of https://github.com/BluBracket/docs.blubracket.com/pull/28


Implement Element.prototype.getElement(s)

I'v implemented following API:

Element.prototype.getElement()
Element.prototype.getElements()

Updates:

implemented table in README

Tests(roughly):

https://github.com/azu/testium/compare/Element.prototype.getElement

Looks good. Thanks!

published in 0.12.0
Please submit a PR for https://github.com/azu/testium/compare/Element.prototype.getElement with the updated published version of webdriver-http-sync.

@EndangeredMassa Thanks for merge!
I'v created pull-request to testium : https://github.com/groupon-testium/testium/pull/129


Code generation failed for method '[FSharp.Core]Microsoft.FSharp.Core.OptimizedClosures' after updating

I've used CoreRt and it works perfectly for some of my F# + Avalonia applications.
But now when I've tried to update one of them I receive the next exception:
EXEC : error : Code generation failed for method '[FSharp.Core]Microsoft.FSharp.Core.OptimizedClosures+Invoke@2712<System.Collections.Generic.KeyValuePair`2<int32,System.__Canon>,int32,System.Collections.Generic.KeyValuePair`2<int32,System.__Canon>>.Invoke(KeyValuePair`2<int32,__Canon>)' [SomeProjects.fsproj]
  ILCompiler.CodeGenerationFailedException: Code generation failed for method '[FSharp.Core]Microsoft.FSharp.Core.OptimizedClosures+Invoke@2712<System.Collections.Generic.KeyValuePair`2<int32,System.__Canon>,int32,System.Collections.Generic.KeyValuePair`2<int32,System.__Canon>>.Invoke(KeyValuePair`2<int32,__Canon>)' ---> System.NotImplementedException: getTailCallHelpers
     at Internal.JitInterface.CorInfoImpl.getTailCallHelpers(CORINFO_RESOLVED_TOKEN& callToken, CORINFO_SIG_INFO* sig, CORINFO_GET_TAILCALL_HELPERS_FLAGS flags, CORINFO_TAILCALL_HELPERS& pResult)
     at Internal.JitInterface.CorInfoImpl._getTailCallHelpers(IntPtr thisHandle, IntPtr* ppException, CORINFO_RESOLVED_TOKEN& callToken, CORINFO_SIG_INFO* sig, CORINFO_GET_TAILCALL_HELPERS_FLAGS flags, CORINFO_TAILCALL_HELPERS& pResult)
     --- End of inner exception stack trace ---
     at Internal.JitInterface.CorInfoImpl.CompileMethodInternal(IMethodNode methodCodeNodeNeedingCode, MethodIL methodIL)
     at Internal.JitInterface.CorInfoImpl.CompileMethod(MethodCodeNode methodCodeNodeNeedingCode, MethodIL methodIL)
     at ILCompiler.RyuJitCompilation.CompileSingleMethod(CorInfoImpl corInfo, MethodCodeNode methodCodeNodeNeedingCode)
     at ILCompiler.RyuJitCompilation.CompileSingleThreaded(List`1 methodsToCompile)
     at ILCompiler.RyuJitCompilation.ComputeDependencyNodeDependencies(List`1 obj)
     at ILCompiler.DependencyAnalysisFramework.DependencyAnalyzer`2.ComputeMarkedNodes()
     at ILCompiler.RyuJitCompilation.CompileInternal(String outputFile, ObjectDumper dumper)
     at ILCompiler.Compilation.ILCompiler.ICompilation.Compile(String outputFile, ObjectDumper dumper)

Last time I've build this app near 3 months ago so I don't remember the exact version of CoreRt.
As a workaround: Is there is a way to get the full list of versions with its dates? Just to specify it instead of 1.0.0-alpha-* inside of fsproj

Tested with
<PackageReference Include="Microsoft.DotNet.ILCompiler" Version="1.0.0-alpha-28410-02" />

and it works.
I'll try to play with newer versions to get when it started to fail.

Was possibly introduced here

https://github.com/dotnet/corert/blob/3a61e443713cfe79ad0a0cef7d2c18c44f7dbd44/src/JitInterface/src/CorInfoImpl.cs#L2599
Related issue: https://github.com/dotnet/runtime/issues/35423

I have submitted #8252 to restore the earlier behavior for CoreRT. It is not 100% correct and there is some chance that it may lead to stack overflow exception with FSharp programs, but it works fairly well in practice.


Don't ensure_table_uri when creating a table with_log_store

Description
Currently when a CreateBuilder is provided with a log store, it will try to create a URL from it's root (via ensure_table_uri), and then re-convert that to a String
https://github.com/delta-io/delta-rs/blob/9264edea89a2fc1c35f4a6b9faab125748ff3651/crates/deltalake-core/src/operations/create.rs#L222-L226
Besides potentially create missing dirs in the case of local FS, I don't think this round trip is actually needed in other cases (e.g. cloud URLs).
The issue this leads to for us is that ensure_table_uri in turn calls resolve_uri_type, which errors out if the URL scheme is not for one of the registered object store factories.
Use Case
I'd like to be able to use with_log_store on the CreateBuilder without having to register an object store factory beforehand.
In other words I'd like to be able to re-use the object/log store that I keep an Arc to without re-creating one from scratch every time.
Related Issue(s)

I guess one resolution to this would be to have my ObjectStoreFactory just return the pre-configured ones according to the URL, which is also fine.

@gruuya I see you're following main closely :laughing: Are you working on a different storage integration, or just concerned with the goofy file creation behavior in core here?

I see you're following main closely

Yeah, pretty nice to see the pace of development here, kudos :)

Are you working on a different storage integration, or just concerned with the goofy file creation behavior in core here?

I was mostly confused by how previously I had an object/log store instance that I'd supply to the CreateBuilder and be done with it, whereas now I need to register the factory as well.
Come to think of it, this mechanism not only makes sense broadly speaking but it's a pretty good idea actually, given that I've been contemplating how to introduce dynamic object/log store creation down the road.
Sorry for the noise and thanks!


How to quickly migrate S3 for local data in Distribution

I initially deployed distribution using local storage in '/var/lib/registry', and then I wanted to switch to s3 storage. How should I migrate local data to s3

The only way to do this is to pull and push with docker cli. Alternatively you can use crane.

skopeo is another option for registry-to-registry copies.


etcd alerting with critical severity on lost watch connection

Recently we've been seeing this:
Oct 17 15:34:53 k8s-master-24469338-0 etcd[3502]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 17 15:34:53 k8s-master-24469338-0 etcd[3502]: message repeated 8 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")]
Oct 18 13:26:06 k8s-master-24469338-0 etcd[124628]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 13:26:07 k8s-master-24469338-0 etcd[124628]: message repeated 62 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")]
Oct 18 14:03:00 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 1; CANCEL")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: message repeated 4 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")]
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: message repeated 2 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")]
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: message repeated 5 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")]
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: message repeated 3 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")]
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: message repeated 2 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")]
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:18:55 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: message repeated 2 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")]
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = client disconnected")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: message repeated 2 times: [ failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")]
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 18 14:35:16 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = body closed by handler")
Oct 18 15:11:56 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 629; CANCEL")
Oct 18 15:44:00 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 273; CANCEL")
Oct 18 16:14:38 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 1017; CANCEL")
Oct 18 17:03:06 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 1217; CANCEL")
Oct 18 19:16:34 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 981; CANCEL")
Oct 18 19:58:34 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 2317; CANCEL")
Oct 18 21:40:18 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 1341; CANCEL")
Oct 19 03:38:02 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 2251; CANCEL")
Oct 19 04:41:46 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 2417; CANCEL")
Oct 19 06:05:36 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 19 06:52:50 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 19 09:51:42 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 19 12:13:14 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 4129; CANCEL")
Oct 19 12:49:48 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 4251; CANCEL")
Oct 19 13:26:42 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 19 13:31:57 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 4387; CANCEL")
Oct 19 16:23:05 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 7955; CANCEL")
Oct 19 17:24:50 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 13583; CANCEL")
Oct 19 18:05:21 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 8101; CANCEL")
Oct 19 20:28:07 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 8273; CANCEL")
Oct 20 00:19:07 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 14559; CANCEL")
Oct 20 01:12:33 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 14687; CANCEL")
Oct 20 04:28:59 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 20 07:33:40 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 15687; CANCEL")
Oct 20 09:04:25 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 9215; CANCEL")
Oct 20 10:03:02 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Canceled desc = context canceled")
Oct 20 11:07:37 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 16215; CANCEL")
Oct 20 11:42:18 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 16303; CANCEL")
Oct 20 13:19:09 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 9531; CANCEL")
Oct 20 14:04:46 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 16621; CANCEL")
Oct 20 14:52:33 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 9661; CANCEL")
Oct 20 16:38:20 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 16999; CANCEL")
Oct 20 19:25:08 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 9993; CANCEL")
Oct 20 22:31:29 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 17913; CANCEL")
Oct 21 02:17:42 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 18451; CANCEL")
Oct 21 03:16:57 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 18561; CANCEL")
Oct 21 04:36:27 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 18799; CANCEL")
Oct 21 05:20:03 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 10761; CANCEL")
Oct 21 06:03:28 k8s-master-24469338-0 etcd[32078]: failed to receive watch request from gRPC stream ("rpc error: code = Unavailable desc = stream error: stream ID 10817; CANCEL")

As per discussion w/ @liggitt @gyuho this seems to be a part from the watch life-cycle, we might want to change this rule and its severity
sum
  by(grpc_service, grpc_method) (rate(grpc_server_handled_total{grpc_code!="OK",job="etcd"}[5m]))
  / sum by(grpc_service, grpc_method) (rate(grpc_server_handled_total{job="etcd"}[5m]))
  > 0.05

cc @brancz

For a few release the etcd alerts are coming from the etcd repository itself.
Can you check the alert there and if necessary open a Issue / PR there? Thanks.
https://github.com/etcd-io/etcd/tree/master/Documentation/etcd-mixin

why change the alert if the error is in another
https://github.com/etcd-io/etcd/issues/10289

This needs to be resolved by etcd itself. Closing here.


Comments at line endings not recognized

Hello,
Thank you for having made this. Though it may seem minor to some, I really enjoy syntax highlighting which helps spot errors, which imho is useful even on simple file formats like requirements.txt.
On current master branch, there appears to be a small bug for the pip requirements grammer , which prevents it from recognizing trailing line comments.
Here's an example (observe the line that contains NOK):
# OK. A full line comment is recognized as such `language-pip`
   #   OK. Another full line comment with preceding whitespace (also recognized)

six
pytest  # NOK. A TRAILING line comment is NOT duely recognized by `language-pip`

# Naturally, it's good that the '#' below (not preceded by whitespace)
# does not signal the start of a comment !
[-e] git://git.example.com/MyProject#egg=MyProject

Below is an excerpt from the pip docmentation relating to the format of 'requirements' files:

A line that begins with # is treated as a comment and ignored.
Whitespace followed by a # causes the # and the remainder of the line to be treated as a comment.

I have put together a tiny pull request that fixes the regular expression, which I will submit shortly. Please consider merging it, or otherwise let me know if you see any issues with it.
As a bonus, there is a second pull request that adds an examples directory with a basic requirements file that demonstrates the issue as well as a couple other files that contain requirements file syntax examples from pip documentation.
If you wish, feel free to merge that one as well, since it helps visually inspecting/testing the grammer as one works on it.
Thanks again!
Cheers,
Tabulo[n]

Thanks, @tabulon. I've merged in your second PR now, but obviously haven't had a chance to push a new release yet. I'll see if I have time to do so tonight.

@tabulon I've just pushed out v2.0.1 containing all of your proposed changes. Please let me know if everything is to your satisfaction, and create a new issue if anything arises.

Thank you for your reactivity @caleb531.
I was able install v2.0.1 via atom, and it works as expected.
Thanks again!


CI test matrix verilator cache fail on new branches

This was recently pushed to dev-public:  https://github.com/chipsalliance/caliptra-rtl/commit/51be55d2a356324f01f85f7cfac2c585e80c777f#diff-81f00368f6e2ec134dea48176d97d18019089c9677b783b531d7e669e8828289
I merged this into dev-goog and ran into an issue:  Latest run
The verilator step that saves the installed directory is failing to save the cache:
Run actions/cache/save@v3
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/caliptra-rtl/caliptra-rtl --files-from manifest.txt --use-compress-program zstdmt
Failed to save: Unable to reserve cache with key verilator-v5.002-f3e6951f0c1d, another job may be creating this cache. More details: Cache already exists. Scope: refs/heads/dev-goog, Key: verilator-v5.002-f3e6951f0c1d, Version: d8d92e0975f28a87cab73a7e22924eff0986e9d4648d3eac429d79abece1e6f7
Warning: Cache save failed.

It's certainly in the right direction, but it looks like we need to move the installation of verilator prior to the matrix generation since the caches seem to be clobbering each other in a branch that is running the CI for the first time
I'll take a quick look at this to unblock development at least on the dev-goog branch

I added a build_tool stage prior to the matrix generation here: https://github.com/chipsalliance/caliptra-rtl/pull/70
I'm now running into an issue where it seems top of tree is broken.  Filed https://github.com/chipsalliance/caliptra-rtl/issues/71


Fix TimedConditionVariable hanging forever when now() > max_blocking_time

I'm targetting the PR to 1.9.x just because it's the branch where I was working on.
The same probably applies to master.
The original code did:
std::chrono::nanoseconds nsecs = max_blocking_time - std::chrono::steady_clock::now();

overflows when max_blocking_time has already passed.
I've carefully checked that that was the cause of the hang I was experimenting (I printed everything and checked values, it was overflowing).
Here backtraces of such a hung (thread 1 and thread 2 are the problem).
Extra comment:
The return type of wait_until when a predicate is not passed should be std::cv_status, and not bool (see here). I'm not fixing that here, and returning false when timeout happens (same was happening before as return value of pthread_cond_timedwait is not 0 when timout happens).
Bottom line: Why not just using std::condition_variable_any? Is because a compiler not supporting it? I see that that's actually being used in not linux systems.

We already have #944 for this. It has a different approach, as it leaves the decission on how to timeout to the native pthread function.
Regarding why we have this, we found out that certain implementations of std::condition_variable_any have an std::mutex inside that is taken without timeout. This can lead to the call to wait_until to be non-deterministic.

Regarding why we have this, we found out that certain implementations of std::condition_variable_any have an std::mutex inside that is taken without timeout. This can lead to the call to wait_until to be non-deterministic.

Now that I think about it, we should only use this when building with STRICT_REALTIME

Didn't see the other PR, which looks better. Can't that be pushed?
This might be the cause of many hangs we have in ros2 CI (I can't confirm that, as the error I was trying to fix was other).

Closing this, as I finally realized my fix didn't solve the issue.
Returning when now > max_blocking_time without releasing the mutex is just wrong.
I also tried https://github.com/eProsima/Fast-RTPS/pull/944, it doesn't work because it's not including  <condition_variable> when std::condition_variable_any is used. Correcting that, it worked perfectly for me.
But if I edit the code to use the custom implementation of TimedConditionVariable (with the #944 fix included), it hungs.
As far as I've tried, it seems that passing past point times is still a problem.
I don't have time for looking at the bug, and I'm happy if #944 gets merged as it solves the problem for ROS2 (we don't use STRICT_REALTIME). But just for let you know, the custom implementation of TimedConditionVariable still have problems.
I can provide detailed steps of how to reproduce the issue if you want. It's not easy to reproduce, as I'm using many custom ros2 branches and I have to run a test more than 100 times to make it hung.
When using std::condition_variable_any, it doesn't have, even if I run the problematic test 2000 times.


Image Pull support with cri-o

NOTE: this is a draft, based on a recent discussion between @haircommander, @magowan , @bpradipt  and @littlejawa - to be improved
Motivation
The initial proposal for the image pull involved relaying the PullImageRequest from kubelet to the agent in the VM, through a new API "PullImage" added to the kata shim proccess.
The shim definition is in the containerd code, and it was made as part of the PoC, in a containerd fork.
(see: https://github.com/confidential-containers/containerd/tree/CC-main)
We are now moving away from this proposal, and other solutions are suggested.
Now all of them rely on features that containerd offers, but are not available in cri-o.
In order to make Confidential Containers work with cri-o, we need to find a solution that can be implemented with cri-o without having to reimplement a large amount of code from containerd.
Possible solutions
The discussion is still in progress. We have two main options at the moment:
PullImage offload
Like in the initial proposal, we could make cri-o relay the PullImage request from kubelet to the agent, by calling the kata shim.
This requires adding the API in the containerd-shimv2 interface (in containerd), and having the corresponding implementation in kata (both of which are already done in fork/branches).
(see: https://github.com/confidential-containers/containerd/blob/CC-main/runtime/v2/task/image.proto and https://github.com/kata-containers/kata-containers/blob/CCv0/src/runtime/pkg/containerd-shim-v2/image.go)
We have a proposal for the required changes in cri-o, but it seems like a big change, and the cri-o community would rather avoid that if we can.
(see: https://github.com/cri-o/cri-o/pull/6724)
Remote Volume
A volume is used by cri-o to pull the image in it, and make it available to the VM.
The guest is responsible for validating the signature for images inside the volume.
This solution is unclear at the moment, and may require some changes on the cri-o side to allow pulling images in a separate volume for confidential containers. But it seems less invasive than trying to make the image pull happen in the guest.
Some concerns on how to validate signatures remain, but the first question to answer is how to make the volume accessible from the guest - this will be our first step of investigation.
Actions
We need to investigate the feasibility/complexity of the remote volume solution (@littlejawa).
This issue will be edited / improved as we progress on it. Any comment / suggestion is welcome.

Thanks @littlejawa, the remote volume approach does have an increased challenge for peer pods over the the pullimage offload for peer pods.
Our key goals to guide us.

Guest can validate the containers
Guest can use artifacts typically available today to verify containers. Nothing special needs done just to use Confidential containers
Improve performance
Something that works for peer pods as well as nested

Observation is that pulling on guest will expose any credentials used  to connect to a registry/repository. View is that confidentiality is not impacted because

container is still verified in guest before use
container image can have encrypted layers which are only available decrypted inside the guest

We should add links to the containerd discussions for completeness


binder link/badge

I didn't see a binder badge in the README, we should fix that.
Not really a Julia expert but from what I can tell you have a REQUIRE file so this should just build on binder now right?

I use the link for each lesson in the table of content (instead of badges). But I can add the badge at the bottom if that's the binder "convention"!

Ah! I naively assumed the link in the TOC just linked inside the repo and didn't even hover it :)

That is so smart! Especially with directly opening the relevant notebook.
Maybe have a badge for old people 👴  like me :)

I'll make the behavior of the links more explicit (and add the badges!).


fix: infinite loop if go.mod not found, warn on missing go.mod or forgotten go get

Fixes https://github.com/a-h/templ/issues/418

I'm wondering if the lack of a go.mod should not be an error. Since they are an optional feature of go.

Although it prints an error if the go.mod is not present, it doesn't actually stop anything @joerdav

Ah yeah, it doesn't exist just a warning log, I forgot about that one! Cool LGTM

One thing, the test project could go in a testdata folder, then it'll only ever be compiled on test runs.


Incorrect Aachen Day-Night result

Hi, I tried to use your pretrained pittsburgh_WPCA4096 model to test on Aachen Day-Night dataset, but got totally incorrect result.
here is my complete process.

generate Aachen_db_path.txt and Aachen_query_path.txt

run the feature_extract.py for db and query images use following two commands.

python feature_extract.py \
  --config_path patchnetvlad/configs/speed.ini \
  --dataset_file_path=Aachen_db_path.txt \
  --output_features_dir patchnetvlad/output_features/aachen_index

python feature_extract.py \
  --config_path patchnetvlad/configs/speed.ini \
  --dataset_file_path=Aachen_query_path.txt \
  --output_features_dir patchnetvlad/output_features/aachen_query

run the 'feture_match.py' for feature matching without providing ground_truth_path

python feature_match.py \
  --config_path patchnetvlad/configs/speed.ini \
  --query_file_path=Aachen_query_path.txt \
  --index_file_path=Aachen_db_path.txt \
  --query_input_features_dir patchnetvlad/output_features/aachen_query \
  --index_input_features_dir patchnetvlad/output_features/aachen_index \
  --result_save_folder patchnetvlad/results/aachen_subset

then process PatchNetVLAD_predictions.txt, for each query image it output top 100 reference images. I just used the pose of top1 reference image as the estimated pose of each query image and post the result to benchmark, but got totally wrong result. here is the result:

I want to know what's wrong with my steps and why it outputs such unacceptable results. Is the model version that I used is wrong? Or have you test your models on Aachen dataset, if you did, could you please provide the result ?

@mingu6 - I remember you wrote the original Aachen script that we used. Any hints?

@mingu6 - I remember you wrote the original Aachen script that we used. Any hints?

Hi, I have solved this problem, you can close this issue, thank you for your reply.

Hi, I have solved this problem, thank you for your reply.

Could you please let us know how you solved it?

the result file PatchNetvlad_predictions.txt is in Kapture Format, it split query and db images by , ,  but I didn't notice it, so the file I submit to benchmark websit is incorrect. just like this:

dusk/rear/1424450493097851.jpg, 0.759246 -0.627745 0.105474 0.135487<PHONE_NUMBER>6856844 -58.480848029973274<PHONE_NUMBER>9472407

So I got totally wrong result which contains all zeros, it seems that if your submit file's format is incorrect, it will output all zeros, just a tips I learned from this issues.


[Auto-generated] Check & update dependencies (pyproject.toml)

Update dependencies (pyproject.toml)
Automatically created PR based on the 'CI - Check pyproject.toml dependencies' workflow in SINTEF/ci-cd.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (ci/dependency-updates@cec69b9). Click here to learn what that means.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@                   Coverage Diff                    @@
##             ci/dependency-updates     #154   +/-   ##
========================================================
  Coverage                         ?   90.71%
========================================================
  Files                            ?       26
  Lines                            ?      431
  Branches                         ?        0
========================================================
  Hits                             ?      391
  Misses                           ?       40
  Partials                         ?        0

Flag
Coverage Δ

linux
90.71% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add storybook/builder-vite

This passes locally when using yarn test storybook --release 2.9.9.

For the upcoming vite-3 beta it would be great if you could set up a separate branch for vite-3 support and add that to your config too. That way you learn about breaking changes early on.
Examples:
https://github.com/vitejs/vite-ecosystem-ci/blob/main/tests/rakkas.ts#L8
https://github.com/vitejs/vite-ecosystem-ci/blob/main/tests/svelte.ts#L8
2 observations from your package.json

no packageManager field means it'll use whatever version of yarn is in path right now which may not be what you want/need
prepublish contains a cd  statement, which alters the current directory. This potentially breaks vite-ecosystem-ci  that uses cd itself and may not expect a command to change it again

If yarn itself figures this out it'll work.
This repo uses ni to detect the package manager and corepack to honor the packageManager field.

@dominikg I've removed the cd commands and targeted a vite-3 branch, as requested.

right now there isn't,  we could set up a 2.9 branch here and a separate scheduled action that uses that branch and the vite 2.9 branch instead of main.
But i don't think we have to as 2.9 won't see another release barring important bug fixes which we could test manually.
cc @patak-dev
Either way this discussion doesn't prevent us from merging. Thank you very much for contributing this!


Hetzner Cloud CCX33 instance machine details

@gunnarmorling could you perhaps post details on this machine. Hetzner info does not include specific CPU, memory configuration incl. bandwidth etc. Would be interesting to determine utilization etc.

Happy to, if I can. Any specific commands I should run whose output you'd like to see?

Not sure what is best on linux. lscpu as a minimum perhaps. hwinfo or similar if possible.
Would if possible in vm like to know specific CPU core/arch, Zen 3?, cache configuration L1/L2 etc, freq, memory configuration (channels, clock, bw).

It would be great if we could see the output of lscpu

Here it is. I.e. EPYC-Milan:
$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         40 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC-Milan Processor
    CPU family:          25
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            1
    BogoMIPS:            4792.79
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt
                          aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xget
                         bv1 xsaves clzero xsaveerptr wbnoinvd arat umip pku ospke rdpid fsrm
Virtualization features:
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   128 KiB (4 instances)
  L1i:                   128 KiB (4 instances)
  L2:                    2 MiB (4 instances)
  L3:                    32 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Vulnerable: Safe RET, no microcode
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

it seems Hetzner uses a mix of Milan and Genoa processors for their "dedicated vCPUs" instances

Ah, that's interesting, where did you get that info from? Might explain the much better numbers which @ebarlas reported after setting up his own CCX33 instance (in another, newer Hetzner DC).

@gunnarmorling
https://www.hetzner.com/cloud in the dedicated vCPU tab:

Optimize your workload with AMD Milan EPYC™ 7003 and AMD Genoa EPYC™ 9654 processors.

Ah, I see. Seems there's no way to find out which one you'd get? Unless
it's unique per DC. Kinda bizarre, definitely an interesting learning for
me from this challenge :)

Message ID: @.***>

Which DC did you rent from?

It seems that Milan has no AVX-512 while Genoa does. Wikipedia claims that AVX-512 on EPYC is available on Zen 4 and later. Milan is Zen 3 and Genoa is Zen 4.

It seems that Milan has no AVX-512 while Genoa does. Wikipedia claims that AVX-512 on EPYC is available on Zen 4 and later. Milan is Zen 3 and Genoa is Zen 4.

AVX-512 is officially supported by Zen 4, look e.g. here: https://www.amd.com/system/files/documents/4th-gen-epyc-processor-architecture-white-paper.pdf (look for 'avx-512' mentions). Zen 3 doesn't support AVX-512, but only AVX2 and below.
Presence of AVX-512 will probably affect performance of all vectorized code (autovectorized and/or manually vectorized using Vector API from Project Panama).
If you have AVX-512-capable CPU then you can measure the difference by running JVM with -XX:UseAVX=2 (or something like that) to limit the AVX level used by JVM (IIRC original AVX=1, AVX2=2, AVX-512=3).

So I am considering to get an AMD EPYC 7401P from the Hetzner Server Auction. That's Zen 1, i.e. I reckon slower per core, but then it has 24 cores :) Like Zen 3, it has AVX2. Numbers wouldn't be comparable of course, but once we've set up hyperfine, it shouldn't be a problem to run all entries again and update the leader board accordingly (apart from the overall absolute shift, there might be relative changes in case different contenders handle the increased core number differently).
My biggest question there is around administering the thing (e.g. how to disable turbo-boost and SMT, which would be a good idea), as I'm not super-savvy when it comes to that.
The alternative would be to re-run everything on the existing instance (which is much faster as of today, no idea why). But I don't feel very confident about it, not being sure whether there might not be a change in performance again. I've also asked the community for help, let's see what comes out of it.
Open for any help and suggestions of course :)

Zen 1 has very high penalties for inter-chiplet communication (actually even inter-CCX communication). Zen 2 brought the central IOD (IO die) and made the inter-chiplet communication much more robust and faster. If you're going for Zen, I would recommend at least Zen 2.
There's Zen 4 available in the form of AX52 server with Ryzen 7 7700: https://www.hetzner.com/dedicated-rootserver/matrix-ax . It has single CCX, so it should be easy to tune multithreading for that chip. Server finder https://www.hetzner.com/dedicated-rootserver show it's "available in few minutes", but direct link to search results (in that server finder) somehow doesn't work.

A dedicated machine is by far the most important here, with min Avx2 support. I am not too worried about cache hierarchy here given the highly parallellizable problem and all solutions doing chunks per processor. However, Zen 1 has some issues with certain simd/Avx2 instructions, high latencies etc. Not sure any such are or will be used here though due to the simple usages. load, cmp, movemask, lzcnt, etc.
disks are not important given entire file cached in memory. the more cores the less difference in efficiency probably. more limited by mem bw/cache.
I don't know how good java AVX-512 support is, but would not see as requirement also harder for most to test locally since many don't have dev machines with it. I don't for example.

There's Zen 4 available in the form of AX52 server with Ryzen 7 7700

@shipilev recommended to use EPYC rather than Ryzen; the reasoning is a bit above my pay grade, though :) There's the AX161 EPYC 7502P (Zen2), though a bits towards the pricier end. Oh the options... .

I would rather take a Zen 3 consumer dedicated machine than a server based
simply on this better matching what developers have at their disposal, this
can quickly become a race for who has access to certain machines or
similar. It already is a diverse set of CPUs out there of course, though.
Thermal throttling due to boosting is an issue on most modern CPUs anyway.
Better to invest in more rigorous and statically sound benchmarking. In
dotnet we would always use BenchmarkDotNet, and forget about process
start/stop.
On Mon, Jan 8, 2024, 22:51 Gunnar Morling @.***> wrote:

There's Zen 4 available in the form of AX52 server with Ryzen 7 7700
@shipilev https://github.com/shipilev recommended to use EPYC rather
than Ryzen; the reasoning is a bit above my pay grade, though :) There's
the AX161 https://www.hetzner.com/dedicated-rootserver/ax161 EPYC 7502P
(Zen2), though a bits towards the pricier end. Oh the options... .
—
Reply to this email directly, view it on GitHub
https://github.com/gunnarmorling/1brc/issues/189#issuecomment-1881877799,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACSMN3Z2HXS2QKLE6ROI3JDYNRS43AVCNFSM6AAAAABBP2DKD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBRHA3TONZZHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

My biggest question there is around administering the thing (e.g. how to disable turbo-boost and SMT, which would be a good idea), as I'm not super-savvy when it comes to that.

I have a script that sets my CPU to max non-turbo frequency:
sudo cpupower frequency-set -g performance
sudo cpupower frequency-set -u 3400MHz -d 3400MHz

Not sure if it would work on others' machines.
As for SMT: why disable it? If someone doesn't want it, then setting CPU affinity mask would amount to the same. Details: https://linux.die.net/man/1/taskset

I don't know how good java AVX-512 support is, but would not see as requirement also harder for most to test locally since many don't have dev machines with it. I don't for example.

As I said before, you can run JVM with -XX:UseAVX=2 and avoid AVX-512-related surprises.
AVX-512 adds support for lane masking and that could potentially allow more interesting programs. But, OTOH, Zen 4 has some bad AVX-512 instructions implementations. https://www.hwcooling.net/en/how-good-is-amds-avx-512-does-it-improve-zen-4-performance/ says:

On the other hand there are some instructions that perform much worse than what is necessitated by the use of 256-bit units and 256bit load/store. That is a case with Compress (vcompressd) operations and Scatter/Gather performance is also poor. Scatter/Gather is also not great on Intel, but to a lesser extent.

As for Ryzen vs Epyc choice: the Ryzen servers that Hetzner provides can have ECC enabled for RAM. The configuration page says:

Upgrade to ECC RAM €4.76 monthly

That should make the server reliable enough for lots of heavy workload hammering :) I'm not sure what improvements Epyc would  bring on top of that.

My biggest question there is around administering the thing (e.g. how to disable turbo-boost and SMT, which would be a good idea), as I'm not super-savvy when it comes to that.

I think we shouldn't disable SMT. Some solutions scale perfectly well with hyperthreading (i.e nearly 2x performance when run at 16 threads on a 8c16t PC).

I agree on not disabling SMT. SMT was used before and workload scales fine
on it.
AX41-NVMe Zen2 3600 or AX52 Zen4 7700 both seem like good enough options.
I'd then exclude AVX-512 usage mainly due to how many devs have access if
zen 4.
On Tue, Jan 9, 2024, 06:10 lehuyduc @.***> wrote:

My biggest question there is around administering the thing (e.g. how to
disable turbo-boost and SMT, which would be a good idea), as I'm not
super-savvy when it comes to that.
I think we shouldn't disable SMT. Some solutions scale perfectly well with
hyperthreading (i.e nearly 2x performance when run at 16 threads on a 8c16t
PC).
—
Reply to this email directly, view it on GitHub
https://github.com/gunnarmorling/1brc/issues/189#issuecomment-1882422469,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACSMN335YL2SG7OLTSG5AELYNTGLHAVCNFSM6AAAAABBP2DKD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBSGQZDENBWHE
.
You are receiving this because you authored the thread.Message ID:
@.***>

Ok, so if we want to stick to Hetzner (which I'd prefer, so as to limit the search space somewhat), it seems AX52 (AMD Ryzen™ 7 7700, Zen 4) would be the best fit. I'm just not sure whether turbo boost can be disabled in that setting? But I'm also not sure how much that may skew results? CC @rschwietzke

I think at the end of the contest, maybe top 10 solutions are selected and run again on a dedicated physical machine (no remote server), with turbo boost disabled, and as few as possible background processes running.

Ha yeah, would love that. Just would need to get my hands on one :)

I think turbo boost should be left on. Thermal throttling is part of the game and rigorous benchmarking will show this, e.g. high variation in results or similar. It's a dedicated DC server, it should have consistent and reliable cooling. Little fluctuation. Handle it by better benchmarking. Make it part of the challenge.
If someone can make a single threaded solution at highest single (or fewer) threaded boost clock that is faster than (all hardware threads) multi threaded lower boost clock then that's game.

I would say off because: It is not that much of a boost for "desktop" CPUs and it might turn the execution order into a factor as well as the CPU usage. Turbo often works only when only a few core are on.
Cloud machines (which is the deployment norm) don't have turbo modes at all.
And yes, we can turn that off for AMD (do that for my notebook sometimes).

Yeah, the motivation for turning it off would be better comparability between different contenders, in particular you'd want to avoid one subsequent run to suffer from being throttled due to a previous run. I suppose one could kinda get on top of it by pausing in between, but that's more voodoo than anything else.

And yes, we can turn that off for AMD (do that for my notebook sometimes).

Do you do that in the BIOS or at OS level? Because I reckon the former isn't available with Hetzner dedicated host (if one only could try out before committing to it...).

OS level. SYSCTL settings.

Alrighty, after confering some more with @rschwietzke and @shipilev, I'm gonna set up an AX161 (AMD EPYC™ 7502P, 32 Core Rome (Zen2)). It's the same in terms of ISA as the original one (i.e. no AVX-512), which also nice. I'll run in 8 cores, as before.

I'm gonna close this one. We've moved to aforementioned instance AX161, and the leader board has been updated to reflect this move.

(..) I'll run in 8 cores, as before.

that means 1 thread per each of the 8 cores? zen has 2 threads per core. is smt disabled now?
the original 8 vcpu cloud machine probably had just 4 cores with 2 vcpu / core = 8 vcpus total. the lscpu for the previous cloud machine said:
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC-Milan Processor
    CPU family:          25
    Model:               1
    Thread(s) per core:  2
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            1
    BogoMIPS:            4792.79
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt
                          aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xget
                         bv1 xsaves clzero xsaveerptr wbnoinvd arat umip pku ospke rdpid fsrm
Virtualization features:
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   128 KiB (4 instances)
  L1i:                   128 KiB (4 instances)
  L2:                    2 MiB (4 instances)
  L3:                    32 MiB (1 instance)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7

it says 4 cores per socket and 1 socket, so 4 cores total, but 2 threads per core = 8 threads total.
also 4 instances of L1 and L2 cache says that there were just 4 cores.

Yes, SMT is disabled, and we run on eight cores out of 32 via numctl. This is the lscpu output of the new machine:
lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         43 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  64
  On-line CPU(s) list:   0-31
  Off-line CPU(s) list:  32-63
Vendor ID:               AuthenticAMD
  Model name:            AMD EPYC 7502P 32-Core Processor
    CPU family:          23
    Model:               49
    Thread(s) per core:  1
    Core(s) per socket:  32
    Socket(s):           1
    Stepping:            0
    Frequency boost:     disabled
    CPU max MHz:         2500.0000
    CPU min MHz:         0.0000
    BogoMIPS:            4990.70
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 mov
                         be popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx
                         2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pa
                         usefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es
Virtualization features:
  Virtualization:        AMD-V
Caches (sum of all):
  L1d:                   1 MiB (32 instances)
  L1i:                   1 MiB (32 instances)
  L2:                    16 MiB (32 instances)
  L3:                    128 MiB (8 instances)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-31
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; untrained return thunk; SMT disabled
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
  Srbds:                 Not affected
  Tsx async abort:       Not affected

I am planning to run the Top 5 or so on all 32 cores (64 threads with SMT) towards the end of the challenge, so as to see how far we can push it below 1 sec :)


Frame pointer ownership?

What's the policy regarding the frame data pointer returned from rs_get_frame_data @ https://github.com/IntelRealSense/librealsense/blob/master/include/librealsense/rs.h#L390 ?
I.e., how long is it valid, who will delete it, ... ?

The data returned by rs_get_frame_data, as well as rs_get_frame_timestamp, is valid until the next call to rs_wait_for_frames, or until rs_stop_device is called or the context itself is destroyed with rs_delete_context.

Thanks - perhaps a bit of background: I'm passing the buffers on to GStreamer, which then basically takes ownership of the pointer and would, at some point, try to free them again. Since I'll have to process the data anyway, I will probably just write it to a newly allocated buffer which GStreamer can then manage on its own.