Dynamic height / scrollbars

The scrollbar is a bit ugly. Is it possible to have the Tree adjust its height dynamically instead of having it fixed and let the parent container handle the scrollbars? Or alternatively be able to style the scrollbars so that it e.g. they are only visible on mouse hover?

I managed to solve this by calculating height based on number of nodes in the tree and adjusting the height of the Tree component accordingly.

I usually make the height of the tree the same height as it's parent using a ResizeObserver on the parent, the passing the width/height to the tree.
If you make the tree height the same height as all the nodes, you will loose the row virtualization that makes it render fast. For example, if you have 10,000 nodes and the tree height is set to 10,000 x rowHeight,  you'll be rendering all 10,000 of them in the tree. Whereas if the height is only 500, you'll only be rendering 20-30.
I would welcome an example where the scrollbar is custom styled though.

@jameskerr do you have a current example with the above? I'm also trying to set the height of the tree dynamically as they expand

Hi @jameskerr Could you please post some code example of how you achieve this? I've been wrestling with this for around 8 hours now, and see some strange results when I try to do this. I've written a few React hooks, tried native JavaScript, but I seem to be get inconsistent results (using ResizeObserver). I actually also get the same inconsistencies when writing regular event listeners on the window.resize as well. Basically, the height of the tree seems to be applied slightly higher than the value ResizeObserver surfaces. And in an inconsistent way. I tried debouncing, but there was still the same issue.
Hence, if you could provide some help with this (via the provision of code for how you use ResizeObserver), that would be appreciated.
Many thanks

Debugging stuff like this is hard. Dang. I’ll give this a more thorough read and post and example, but I’ve had to make the parent component have styles like min-height: 0 and flex: 1

hey @jameskerr  can you provide an example, better in documentation.

I would also love some better documentation on this.

It would be really nice to have the possibility to send a maxHeight and then for the table to be of height auto otherwise

Is it posible to use the height of the parent. I am not setting any height but my issue is that there is a lot of space at the bottom of the tree.
Example my parent height is 252 however the tree is 500. Yes I could set the heigh manually but my parent window is drainable and resizable.
parent: PanelWindow.tsx
return (
    <Draggable
      handle=".handle"
      grid={[25, 25]}
      scale={1}
      defaultPosition={{ x: positionX, y: positionY }}
    >

        {isVisible && (
          <ResizableBox
            style={{ height: initialHeight, width: initialWidth }}
            width={width}
            height={height}
            minConstraints={[250, 250]}
            maxConstraints={[window.innerWidth * 0.9, window.innerHeight * 0.9]}
            resizeHandles={["se", "s", "e"]}
            onResize={handleResize}
          >
            <Card style={{ width, height, boxSizing: "content-box" }}>
              <CardDescription className="handle text-center" ref={headerRef}>
                <XCircle
                  className="absolute top-0 left-0"
                  size={"15px"}
                  onClick={handleDismiss}
                />
                {panelTitle}
              </CardDescription>
              <div
                style={{
                  height: `calc(100% - ${headerHeight}px)`,
                  overflow: "auto",
                }}
              >
                {children}

            </Card>
          </ResizableBox>
        )}

    </Draggable>
  );

ScenePanel.tsx
function ScenePanel() {
  const viewRef = useRef<HTMLDivElement>(null);
  const getViewHeight = () => viewRef.current?.clientHeight || 0;

  useEffect(() => {
    // You can now use getViewHeight() to get the height of the view
    const height = getViewHeight();
    console.log("View height:", height);
  }, []);

  return (

      <PanelWindow
        isVisible={true}
        className="scene"
        panelTitle="Scenes"
        onDismiss={() => {}}
        positionY={0}
        positionX={0}
      >

          <Tree initialData={demoData}>{Node}</Tree>

      </PanelWindow>

  );
}

The ScenePanel size is showing 500 while the parent is only 252.

Yes, I usually make the parent expand to fill it's parent (either with flex: 1 or the css below).
height: 100%;
min-height: 0;

Then I add a ref to it with this package https://github.com/ZeeCoder/use-resize-observer
That hook will return the height and width of the parent whenever it changes. You then pass these numbers to the Tree.
const { ref, width, height } = useResizeObserver<HTMLDivElement>();

  <Tree height={height} width={width} />

Also having some trouble getting this to work the way I want. Is it possible to disable scrolling for the tree and have another component take over that role? For example we have a ScrollArea implementation from Radix UI that we could use. Or would this screw with the virtualization?
I have tried getting the same behavior as the Cities example, where horizontal scrolling is disabled and text has ellipsis overflow instead by using the FillFlexPanel, but I cannot get the horizontal scrollbar to be hidden with overflow-x: "hidden" since the root element of the tree seems to have overflow: "auto" hard-coded into it?

Ok so in that case, you'll need to set overflow hidden on your Node renderer. Handle the ellipses in the Node Renderer as well.
The tree component should fill its container, and the tree wrapper should have overflow scroll, but then the nodes should be width: 100% with overflow: hidden.
I've never used custom scrollbars in my app, so if you get it working and want to throw up an example, I could publish that in an FAQ somewhere.

It looks like it could be possible to use custom scrollbars if the outerElementType of the react-window implementation was able to be swapped out: https://github.com/bvaughn/react-window/issues/110#issuecomment-469061213.
In the example they are using react-custom-scrollbars so I don't see why another component such as ScrollArea from Radix shouldn't work.

Cool! Thanks for researching that.

Да, я обычно расширяю родительский элемент, чтобы заполнить его (либо с помощью flex: 1, либо с помощью CSS ниже).
height: 100%;
min-height: 0;

Затем я добавляю к нему ссылку с помощью этого пакета https://github.com/ZeeCoder/use-resize-observer .
Этот крючок будет возвращать высоту и ширину родительского элемента всякий раз, когда он изменяется. Затем вы передаете эти числа в Дерево.
const { ref, width, height } = useResizeObserver();

  <Tree height={height} width={width} />

I used your example as a solution to the problem, but the height of the tree is still 500px (I think this is the default)
const {ref, height } = useResizeObserver()

return (

                <Tree data={structure}
                    indent={24}
                    height={height}
                    width="auto"
                    rowHeight={66}
                    idAccessor='uuid'
                >
                    {children}
                </Tree>

    )

You'll need to make sure that the parent is filling its container. It needs to have flex: 1; on it or grid-template-row 1fr or something to make it fill the container.

Yes, I tried using flex, but it also doesn't give any result. I tried to reproduce this code in codesandbox to make sure on whose side the problem is. Maybe I don't understand correctly how flex works
https://codesandbox.io/s/nostalgic-carlos-2l38qd?file=/src/App.js

For those using Tailwind, I was able to apply custom styles to the scrollbar with this nifty Tailwind plugin: https://github.com/adoxography/tailwind-scrollbar
Add the Tailwind classes to the  component

I had issues using useResizeObserver where it would resize the height down the way, but if you resized back up it wouldn't fire the resize events. Managed to fix it by setting  height = {height -1 }  in the tree component. the - 1 was just enough to stop the tree from locking the height!

I did get dynamic height working but it wasn't easy.
The principal is that you want:

The list to expand to fit the contents of the tree.
but to not exceed the contents of some container it is contained within.
The virtual list inside to continue to fill all available space.

Essentially we want our tree height to be equal to min(container height, total calculated tree height).
One limitation that is difficult to get around is that we cannot access the calculated tree height easily without changing react-arborist's code. However we can get to it by polling for it within the Tree component - unfortunately there is no other good way to figure out WHEN this becomes available on the DOM. This would be a lot easier with code that let's you subscribe to changes on the total calculated tree height!
I will extrapolate from my working version to show the principals at work:
First set up the DOM & some refs.
const screenRef = useRef(); // where you want to fill the available space of
const containerRef = useRef(); // A div immediately above the tree (may be able to skip this actually (?) and do everything off the screenRef)
const [treeHeight, setTreeHeight] = useState();
<div css={css`height: 100vh; display: flex; flex-direction: column}` ref={screenRef}>
  <div css={css`height: fit-content;`} ref={containerRef}>
    <Tree height={treeHeight} data={...}>

Then let's introduce the polling code.
// 500 = arbitrary starting height
const [treeContentsHeight, setTreeContentsHeight] = useState(500);
const [screenHeight, setScreenHeight] = useState(500);
useEffect(() => {
    // This observer
    const observer = new ResizeObserver(entries => {
      const height = entries[0].contentRect.height;
      setTreeContentsHeight(height);
    });
    // This tree has the correct total height calculated so we want to watch it using ResizeObserver
    let innerTreeElement;
    let pollId;
    const poll = () => {
      pollId = setTimeout(() => {
        innerTreeElement = containerRef.current?.firstChild?.firstChild?.firstChild;
        if (innerTreeElement) {
          clearTimeout(pollId);
          observer.observe(innerTreeElement);
        } else {
          pollId = poll();
        }
      }, 30);
    };
    poll();
    return () => {
      observer.disconnect();
      clearTimeout(pollId);
    };
  }, []);

Finally, whenever the treeContentsHeight is updated, we can set the containerHeight, and make sure it's updated on resize as well.
  useEffect(() => {
    const resizeContainer = (): void => {
      const screenRefHeight = screenRefHeight.current.getBoundingClientRect().height;
      setTreeHeight(Math.min(screenRefHeight, treeContentsHeight));
    };
    resizeContainer();
    window.addEventListener('resize', resizeContainer);
    return () => window.removeEventListener('resize', resizeContainer);
  }, [treeContentsHeight]);

Мне удалось заставить работать динамическую высоту, но это было непросто.
Главное, что вы хотите:

Список, который нужно расширить, чтобы он соответствовал содержимому дерева.
но не превышать содержимое какого-либо контейнера, внутри которого оно содержится.
Виртуальный список внутри продолжит заполнять все доступное пространство.

По сути, мы хотим, чтобы высота нашего дерева была равна min(container height, total calculated tree height).
Одно из ограничений, которое трудно обойти, заключается в том, что мы не можем легко получить доступ к вычисленной высоте дерева без изменения кода реакции-арбориста. Однако мы можем добраться до него, опросив его в компоненте «Дерево» — к сожалению, нет другого хорошего способа выяснить, КОГДА это станет доступно в DOM. Это было бы намного проще с кодом, который позволял бы вам подписываться на изменения общей рассчитанной высоты дерева!
Я экстраполирую свою рабочую версию, чтобы показать принципы работы:
Сначала настройте DOM и несколько ссылок.
const screenRef = useRef(); // where you want to fill the available space of
const containerRef = useRef(); // A div immediately above the tree (may be able to skip this actually (?) and do everything off the screenRef)
const [treeHeight, setTreeHeight] = useState();
<div css={css`height: 100vh; display: flex; flex-direction: column}` ref={screenRef}>
  <div css={css`height: fit-content;`} ref={containerRef}>
    <Tree height={treeHeight} data={...}>

Тогда давайте представим код опроса.
// 500 = arbitrary starting height
const [treeContentsHeight, setTreeContentsHeight] = useState(500);
const [screenHeight, setScreenHeight] = useState(500);
useEffect(() => {
    // This observer
    const observer = new ResizeObserver(entries => {
      const height = entries[0].contentRect.height;
      setTreeContentsHeight(height);
    });
    // This tree has the correct total height calculated so we want to watch it using ResizeObserver
    let innerTreeElement;
    let pollId;
    const poll = () => {
      pollId = setTimeout(() => {
        innerTreeElement = containerRef.current?.firstChild?.firstChild?.firstChild;
        if (innerTreeElement) {
          clearTimeout(pollId);
          observer.observe(innerTreeElement);
        } else {
          pollId = poll();
        }
      }, 30);
    };
    poll();
    return () => {
      observer.disconnect();
      clearTimeout(pollId);
    };
  }, []);

Наконец, всякий раз, когда TreeContentsHeight обновляется, мы можем установитьContainerHeight и убедиться, что он также обновляется при изменении размера.
  useEffect(() => {
    const resizeContainer = (): void => {
      const screenRefHeight = screenRefHeight.current.getBoundingClientRect().height;
      setTreeHeight(Math.min(screenRefHeight, treeContentsHeight));
    };
    resizeContainer();
    window.addEventListener('resize', resizeContainer);
    return () => window.removeEventListener('resize', resizeContainer);
  }, [treeContentsHeight]);

You are simply a genius! Thank you, your method helped me implement the dynamic height of the tree container.
In my case, height 100vh was drawing extra space after Tree , but I set height to treeContentsHeight and the height actually became dynamic


ColorPaletteResources doesn't apply to custom control

Describe the bug
Hi, I've tried to sort out the ColorPaletteResources and how it can be used in theming of the custom control. I found that the palette's accent color doesn't apply to brushes of control where used SystemAccentColor. For example <SolidColorBrush x:Key="CustomControl1BackgroundBrushFocused" Color="{ThemeResource SystemAccentColor}"/> . It works perfectly for standard controls (TextBox) and standalone brushes (which I use for a border) but doesn't for custom control with style in Generic.xaml.
Steps to reproduce
Project to reproduce ColorPalettesTest.zip
MainPage.xaml
<Page
    x:Class="ColorPalettesTest.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:ColorPalettesTest"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d"
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">

    <Page.Resources>
        <ResourceDictionary>
            <ResourceDictionary.MergedDictionaries>
                <ColorPaletteResources Accent="Red"/>
            </ResourceDictionary.MergedDictionaries>
            <SolidColorBrush x:Key="MyAccentBrush" Color="{StaticResource SystemAccentColor}"/>
        </ResourceDictionary>
    </Page.Resources>

    <StackPanel Orientation="Vertical" HorizontalAlignment="Center" VerticalAlignment="Center">

        <TextBox Text="TestText" Margin="5"/>

        <Border Width="100" Height="100" Margin="5"
                Background="{StaticResource MyAccentBrush}">
            <TextBlock Text="It's a border"/>
        </Border>

        <local:CustomControl1 Width="100" Height="100" Margin="5"/>
    </StackPanel>
</Page>

Generic.xaml
<ResourceDictionary
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:ColorPalettesTest">

    <ResourceDictionary.ThemeDictionaries>
        <ResourceDictionary x:Key="Default">
            <SolidColorBrush x:Key="CustomControl1BackgroundBrushFocused" Color="{ThemeResource SystemAccentColor}"/>
        </ResourceDictionary>
    </ResourceDictionary.ThemeDictionaries>

    <Style TargetType="local:CustomControl1" >
        <Setter Property="Background" Value="{ThemeResource CustomControl1BackgroundBrushFocused}"/>
        <Setter Property="Template">
            <Setter.Value>
                <ControlTemplate TargetType="local:CustomControl1">
                    <Border
                        Background="{TemplateBinding Background}"
                        BorderBrush="{TemplateBinding BorderBrush}"
                        BorderThickness="{TemplateBinding BorderThickness}">
                    </Border>
                </ControlTemplate>
            </Setter.Value>
        </Setter>
    </Style>
</ResourceDictionary>

Expected behavior
The ColorPaletteResources should apply to the custom control.
Screenshots

Version Info

NuGet package version:
[Microsoft.WinUI 3.0.0-preview3.201113.0]

Windows app type:

UWP
Win32

Yes

Windows 10 version
Saw the problem?

Insider Build (xxxxx)

May 2020 Update (19041)
Yes

November 2019 Update (18363)

May 2019 Update (18362)

October 2018 Update (17763)

April 2018 Update (17134)

Fall Creators Update (16299)

Creators Update (15063)

Device form factor
Saw the problem?

Desktop
Yes

Xbox

Surface Hub

IoT

Additional context

I'm surprised this works at all, in the code you've posted CustomControl1BackgroundBrushFocused is defind only for the default (dark) theme. Your app looks like its in light theme so I would expect the system to have failed to find the CustomControl1BackgroundBrushFocused resource.  @kikisaints and @MikeHillberg ?

I'm surprised this works at all, in the code you've posted CustomControl1BackgroundBrushFocused is defind only for the default (dark) theme. Your app looks like its in light theme so I would expect the system to have failed to find the CustomControl1BackgroundBrushFocused resource. @kikisaints and @MikeHillberg ?

Perhaps the Default theme is used as a fallback, and that is why it works, whilst not working.
But yes, values should be provided for Default, Light, and HighContrast themes to be as compatible as possible

Thanks for the quick answers. I've modified my sample by your recommendations and it still not working.
Generic.xaml
    <ResourceDictionary.ThemeDictionaries>
        <ResourceDictionary x:Key="Default">
            <SolidColorBrush x:Key="CustomControl1BackgroundBrushFocused" Color="{ThemeResource SystemAccentColor}"/>
        </ResourceDictionary>
        <ResourceDictionary x:Key="Light">
            <SolidColorBrush x:Key="CustomControl1BackgroundBrushFocused" Color="{ThemeResource SystemAccentColor}"/>
        </ResourceDictionary>
        <ResourceDictionary x:Key="HighContrast">
            <SolidColorBrush x:Key="CustomControl1BackgroundBrushFocused" Color="{ThemeResource SystemAccentColor}"/>
        </ResourceDictionary>
    </ResourceDictionary.ThemeDictionaries>

    <Style TargetType="local:CustomControl1" >
        <Setter Property="Background" Value="{ThemeResource CustomControl1BackgroundBrushFocused}"/>
        <Setter Property="Template">
            <Setter.Value>
                <ControlTemplate TargetType="local:CustomControl1">
                    <Border
                        Background="{TemplateBinding Background}"
                        BorderBrush="{TemplateBinding BorderBrush}"
                        BorderThickness="{TemplateBinding BorderThickness}">
                    </Border>
                </ControlTemplate>
            </Setter.Value>
        </Setter>
    </Style>

MainPage.xaml
...
    <Page.Resources>
        <ColorPaletteResources x:Key="Default" Accent="Red" BaseLow="Black"/>
    </Page.Resources>
    <StackPanel Orientation="Vertical" HorizontalAlignment="Center" VerticalAlignment="Center">

        <Button Content="Change Palette" Click="myButton_Click"/>
        <TextBox Text="TestText" Margin="5"/>
        <local:CustomControl1 Width="100" Height="100"/>

    </StackPanel>

Is the x:Key="Default" dark theme? I supposed that is a Light and that if there are no specified resources for other themes (Dark, HighContrast) the Default resources will use instead. At now, it's not clear. Do you have any guide or any docs for the best practices to theming of the custom controls?
Also, for the previous sample (with Default theme only) the palette works perfectly if I place it App.xaml:
<Application.Resources>
        <ResourceDictionary>
            <ResourceDictionary.MergedDictionaries>
                <XamlControlsResources xmlns="using:Microsoft.UI.Xaml.Controls" />
            </ResourceDictionary.MergedDictionaries>
            <ResourceDictionary.ThemeDictionaries>
                <ResourceDictionary x:Key="Default">
                    <ResourceDictionary.MergedDictionaries>
                        <ColorPaletteResources Accent="Red" BaseLow="Black"/>
                    </ResourceDictionary.MergedDictionaries>
                </ResourceDictionary>
            </ResourceDictionary.ThemeDictionaries>
        </ResourceDictionary>
    </Application.Resources>

ColorPalettesTest.zip

Any updates? Could we expect the correctly working of the ColorPalettes to use for our themes engine?

Any updates? Could we expect the correctly working of the ColorPalettes to use for our themes engine?

@alexdi220 do you know if this issue is unique to Winui3 or it it occurs with system xaml (winui2) as well?

@alexdi220 do you know if this issue is unique to Winui3 or it it occurs with system xaml (winui2) as well?

@StephenLPeters I didn't check it on WinUI 2. So I don't have an answer.

@StephenLPeters I didn't check it on WinUI 2. So I don't have an answer.

@kikisaints FYI.

@kikisaints FYI.

I observe a similar behavior when using the theme resources in the code.
<ResourceDictionary>
    <ResourceDictionary.ThemeDictionaries>
        <ResourceDictionary x:Key="Dark">
            <SolidColorBrush x:Key="TitleBarFocusColorBrush" Color="{ThemeResource SystemAccentColorDark3}"/>
      </ResourceDictionary>
        <ResourceDictionary x:Key="Light">
            <SolidColorBrush x:Key="TitleBarFocusColorBrush" Color="{ThemeResource SystemAccentColorLight3}"/>
        </ResourceDictionary>
    </ResourceDictionary.ThemeDictionaries>
</ResourceDictionary>

I always will get the TitleBarFocusColorBrush of the Light definition independent of the sequence of definition or the app theme configuration. The following code is executed multiple times as triggered by a focus event.
	winrt::Microsoft::UI::Xaml::Media::Brush OperationEdit::FocusColor() const
	{
		// Currently we can not use {Themeresource ...} to set a propery value
		// as there seems no set change on the property.
		const auto resources = winrt::Microsoft::UI::Xaml::Application::Current().Resources();
		if (resources.HasKey(winrt::box_value(L"TitleBarFocusColorBrush")))
		{
			winrt::Windows::Foundation::IInspectable color = resources.Lookup(winrt::box_value(L"TitleBarFocusColorBrush"));
			return winrt::unbox_value<winrt::Microsoft::UI::Xaml::Media::SolidColorBrush>(color);
		}
		else
		{
			// This code is just a fall-back (is never executed)
                        return winrt::unbox_value<winrt::Microsoft::UI::Xaml::Media::Brush>(GetValue(FocusColorProperty()));
		}
	}

Originally, I've planned to use the FocusColor() property with a XAML definition {Themeresource ...}. But it seems this will never trigger a property change (never hit a breakpoint). Instead, I tried to extract the proper brush from the resource directories directly. But in this case, I will always get the Light brush color.
Are there any updates on the ticket status?


Set ROCKSDB Engine for MySQL Table Migration

Hi, I am trying to use this tool to create a database in MySQL. But, I want to use ROCKSDB as engine. I couldn't find any setting to set the engine when running migration.
GORM Does it like this - db.Set("gorm:table_options", "ENGINE=InnoDB").AutoMigrate(&User{})
https://gorm.io/docs/migration.html#content-inner

Hey @dev681999,
We don't support this option atm, but it's really simple to add. I can do it in the upcoming days, but if you want to contribute this change, here's where it needs to be added:
1. Add another field named Options to the entsql annotation:
https://github.com/facebook/ent/blob/dea48a78eae387eaa7845480e6a3559029684a14/dialect/entsql/annotation.go#L11-L37
2. Append this annotation option to the table builder as follows:
if opts := t.Annotation.Options; opts != "" {
	b.Options(opts)
}

If @dev681999 can't, i'd be glad to work on this :)

You're welcome to add this @dooven


Custom ExceptionMapper not invoked when Hibernate rollback

Using DW 0.8.0, Java 8
I have one custom exception mapper to handle hibernate exception:
import org.hibernate.exception.ConstraintViolationException;

@Provider
public class SQLConstraintViolationExceptionMapper implements ExceptionMapper<ConstraintViolationException> {
    ....
}

I registered the mapper https://github.com/paukiatwee/budgetapp/blob/master/src/main/java/io/budgetapp/BudgetApplication.java#L143
When org.hibernate.exception.ConstraintViolationException exception is thrown when foreign key violation occurred, SQLConstraintViolationExceptionMapper is no called.
You can reproduce the problem by running this integration test https://github.com/paukiatwee/budgetapp/blob/master/src/test/java/io/budgetapp/resource/CategoryResourceIT.java#L65
This might caused by https://github.com/dropwizard/dropwizard/blob/master/dropwizard-hibernate/src/main/java/io/dropwizard/hibernate/UnitOfWorkApplicationListener.java#L87 exception not handled by container.
Working fine for DW 0.7.0

I can confirm this issue.
Unfortunately, I can't give a solution now. The problem lies in the update to Jersey 2 in 0.8.0.
Seems to be UnitOfWorkApplicationListener doesn't work with exception mappers.

I've made another look and can give a workaround for you.
In 0.8.0 in UnitOfWorkApplicationListener we commit a Hibernate transaction after a request has been processed and response processing has started. We don't have an event like BEFORE_RESOURCE_METHOD_FINISHED, so we are forced to make a commit (with possible errors) during response processing.
In the Jersey architecture at this stage exception mappers are not taken into account, but mappers that implement the interface ResponseErrorMapper are.
They are also not processed by default, so you need to enable this feature in configuration:
environment.jersey().property(ServerProperties.PROCESSING_RESPONSE_ERRORS_ENABLED, true);

After that you can refactor your exception mapper something like that:
@Provider
public class SQLConstraintViolationExceptionMapper implements ResponseErrorMapper {

    private static final Logger LOGGER = LoggerFactory.getLogger(SQLConstraintViolationExceptionMapper.class);

    @Override
    public Response toResponse(Throwable e) {
        return Response.status(Response.Status.BAD_REQUEST)
                .entity(Collections.singletonMap("errors",
                        Collections.singletonMap("message", getMessage((ConstraintViolationException) e))))
                .build();
    }

    private List<String> getMessage(ConstraintViolationException e) {

        LOGGER.error(e.getMessage(), e);

        // a hack to convert exception to friendly error message
        if ("fk_budgets_categories".equalsIgnoreCase(e.getConstraintName())) {
            return Collections.singletonList("Failed to delete category due to references to existing budget(s).");
        } else if ("fk_transactions_budgets".equalsIgnoreCase(e.getConstraintName())) {
            return Collections.singletonList("Failed to delete budget due to references to existing transaction(s).");
        }
        return Collections.singletonList(e.getMessage());
    }
}

Could you test this approach? If it works for you, I will add information about response error mappers
into the 0.8.0 migration guide.

The issue is fixed after update based on your comment commit.
However, I think the proper way to fix this is to rethrow exception as org.glassfish.jersey.server.internal.process.MappableException as the doc mention:
 * A runtime exception that contains a cause, a checked or runtime exception,
 * that may be mapped to a {@link javax.ws.rs.core.Response} instance.
 * <p>
 * The runtime will catch such exceptions and attempt to map the cause
 * exception to a registered {@link javax.ws.rs.ext.ExceptionMapper} that
 * provides an appropriate {@link javax.ws.rs.core.Response} instance.

That actually seems like a better solution.  Jersey will rethrow the original exception if it's umapped and map to Response if it's mapped. Thanks for pointers.
I will try to create a PR for fixing that.


Doc update for stats module

Codecov Report

Merging #83 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #83   +/-   ##
=======================================
  Coverage   90.05%   90.05%
=======================================
  Files         125      125
  Lines        2837     2837
=======================================
  Hits         2555     2555
  Misses        282      282

Impacted Files
Coverage Δ

creme/stats/rolling_quantile.py
94.11% <ø> (ø)
:arrow_up:

creme/stats/variance.py
93.75% <ø> (ø)
:arrow_up:

creme/stats/n_unique.py
93.1% <ø> (ø)
:arrow_up:

creme/stats/ewmean.py
91.66% <ø> (ø)
:arrow_up:

creme/stats/rolling_ptp.py
93.75% <ø> (ø)
:arrow_up:

creme/stats/ptp.py
93.33% <ø> (ø)
:arrow_up:

creme/stats/max.py
92.3% <ø> (ø)
:arrow_up:

creme/stats/kurtosis.py
95.45% <ø> (ø)
:arrow_up:

creme/stats/mode.py
93.33% <ø> (ø)
:arrow_up:

creme/stats/rolling_mode.py
95% <ø> (ø)
:arrow_up:

... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c021ba...e74f00d. Read the comment docs.


Revert "Adding Sprint 1 features from teams: hendrik, tobi (#106)"

This reverts commit 439a475d5d7d95250e9262d9383c210eb965b7f1.

Changes Unknown when pulling 5415f6b5706fc678f564827c25541b941879198e on sprint1_revert into ** on master**.


PDFKit not applying the specified 'css' stylesheet.

I have a html page displaying some images from Facebook CDN which needs to be converted to pdf as is. I am using ruby (2.2.0), rails (4.2.1), pdfkit(0.6.2) and have installed wkhtmltopdf(0.9.9) to the local box using 'sudo apt-get install'.
I precompiled the assets and copied the public/assets/application-xx.css to app/assets/stylesheets/pdf.css. Have replaced the asset urls in the css with absolute path (from /home/prasad/...). Have confirmed that all the referenced assets(fonts & images) are accessible by referring each them in browser url and making sure that they are downloadable.
The problem is that the pdf is being generated but without the css being applied. The images from the CDN are being displayed.
Started GET "/home/generate_pdf.pdf" for <IP_ADDRESS> at 2015-04-10 18:20:25 +0530
Processing by HomeController#generate_pdf as PDF
  Rendered home/generate_pdf.html.haml within layouts/application (9.2ms)
Loading page (1/2)
[============================================================] 100%
Printing pages (2/2)
Done
  Rendered text template (0.0ms)
Sent data booklet.pdf (23.2ms)
Completed 200 OK in 61753ms (Views: 23.0ms)

Have posted the same on Stackoverflow but couldn't find the solution hence have posted it here. https://github.com/prasadsurase/topdf.git is a sample repo reproducing the issue.

Hi @prasadsurase ,
I was able to get the stylesheets to display fine for me.

I did notice a couple roadblocks with getting wkhtmltopdf to work with the application:

A couple image links were broken
wkhtmltopdf demands full links to CSS & JS, so I just commented out those lines in the layout/application.html.haml file. Since you are including the CSS in the controller, this doesn't matter for this particular issue.

I am closing this issue, but feel free to update with additional information and I will reopen.


[Yunikorn-1353] refactor annotation and label processing

What is this PR for?
Annotation and label processing is split over multiple files in the shim code.
Some are in common/utils, some in common/gang_utils. In other places we directly pull them from the pod. Annotations and label processing should move to a centralised place.
Generally: annotations should be used for all additional info we need inside YuniKorn. Labels are not the correct objects from a K8s perspective.
What type of PR is it?

[ ] - Bug Fix
[x] - Improvement
[ ] - Feature
[ ] - Documentation
[ ] - Hot Fix
[x] - Refactoring

Todos

[ ] - Task

What is the Jira issue?

Open an issue on Jira https://issues.apache.org/jira/browse/YUNIKORN/
Put link here, and add [YUNIKORN-Jira number] in PR title, eg. [YUNIKORN-2] Gang scheduling interface parameters

How should this be tested?
Screenshots (if appropriate)
Questions:

[ ] - The licenses files need update.
[ ] - There is breaking changes for older versions.
[ ] - It needs documentation.

@zhuqi-lucas please rebase on the latest master

@0yukali0 can you rebase based on the latest master? There's been significant changes that are causing conflicts.

Thank you, @pbacsko @0yukali0  for review, addressed in latest PR.

recheck

Changes are looking good. Please rebase to the latest so we can commit.

Thank you @wilfred-s for review, rebased it now.

Hi @pbacsko , if you have any other suggestions, thanks!

+1 I'll merge this shortly.


Is it possible to combine 2 gofpdf.fpdf documents?

I am using gofpdf to output structural calculations that can sometimes be more than 100 pages Long. I used to write these documents by hand in a word processor with a little help from Excel files for the calculations. I now have a little console program written in GO that does this much better for me with gofpdf.
I always have to combine 2 fpdfs into 1 document with pdf-merge after they are finished. It is not possible to do this type of document in one loop from beginning to end because of the changing size of the first document while writing the second document. So I merge them. It is still much easier than the word processor.
However I am wondering about combining these 2 fpdfs directly in the program before writing them in just 1 file to disk.
I am seeing some issues here with gofpdi. Is it possible to do this with gofpdi?
Maybe an AddRange function or something like it so that the accumulated bookmarks in the second fpdf would also be added correctly to the bookmarks in the first fpdf.
I am not talking about reading 2 pdf files and combining them. I want to combine 2 fpdfs one after the other!
Is this possible?

Any comments, @phpdave11?

I did actually look up the Adobe pdf spec to try and understand how pdfs are made up. I also looked through your code to see if I could understand it enough to at least visualize a solution. I am a bit overwhelmed by the whole thing. I can easily imagine that there are many things to keep track of in just adding another fpdf-document to the end of the initial fpdf-document.
I am also a thankful user of this package! It saves untold amounts of work.
If you have any ideas… that would be great.
I thought after reading the other issues about gofpdi listed here in gofpdf that converting a fpdf document to a template and then adding it in gofpdi would perhaps be the way of least resistance.
I am also calculating and writing the fpdfs to byte-arrays in a server and then sending them back to the client through gRPC where they are then written to disk. I travel a lot and only want to take a thin client with me to work on the road.
Maybe a function that combines 2 fpdfs in gofpdi is really the best solution.
Thanks again.

I always have to combine 2 fpdfs into 1 document with pdf-merge after they are finished. It is not possible to do this type of document in one loop from beginning to end because of the changing size of the first document while writing the second document.

Is it possible to build up two dynamic, non-PDF data structures during the construction phase? They would contain enough information to generate the PDFs. After this stage, the combined PDF could be generated in a single pass with no special logic needed for bookmark and page number calculations.

The page numbers are the problem.
I need several lists in the first document that are based on the number of items of certain types in the second document: Illustrations, Calculations, Tables, Diagrams, etc. The lists need the page numbers where to find each item.
The first document is just the front matter with the general table of contents and all the lists.
This first document doesn't even get page numbers at all.
It is a minimum of 2 pages long and can be 5 pages long or more.
I start numbering the pages in the second document.
Hold on tight… it gets worse! :-)
The results of the earlier calculations in the second document are used in the calculations later in the document. It is one huge loop of calculations passing the results along from beginning to end.
Thats why I just keep adding pages to the fpdf as I go through the calculations from the beginning to the end of the second document. Along the way I gather the lists of items for the front matter and write the first fpdf document at the end!
It works great!!! I end up with 2 documents: frontmatter.pdf and maincontent.pdf
I only have to merge the 2 documents at the end. I am NOT complaining about this small step. I am already saving myself lots of work with it just as it is.
However...
It sounds like my feature request is an unusual case and probably a lot of work to implement.
If I am the only one that wants it, then it can't become a priority.
I really appreciate you reading this, thankyou!
As a side note: A lot of english technical books are written this way. I just don't have the time to do it in LateX. Maybe this can be a future feature of godpdf.

It may be that aliases could help you. Have you looked at the RegisterAlias example?

That looks very interesting!!!
Is it possible to create more than one table of contents like this in 1 document?
Each of my lists is actually a table of contents.

Is it possible to create more than one table of contents like this in 1 document?

Certainly! You just need to come up with a unique text key that will be replaced, when the document is rendered, with its corresponding text value.

In the examples I see {mark %d}.
Can I use {content %d} for contents and {diagram %d} for diagrams to keep the lists separate??

Yes. You should use any text that will be recognized by the search-and-replace operation that takes place when the PDF is actually written. The UTF-8 version of the example shows different alias text. Even the curly braces are not necessary -- you just have to choose keys that won't otherwise occur in your text. (Otherwise some unexpected replacements would be made.) I hope this scheme works for you.

I need to study the examples closely!!
Thankyou very much!!
There is something similar to this in LateX.
I just need to be careful about it.
We probably can close this issue as solved.
Or we can leave it  for someone who might need it.
Have a nice evening!


Fix copyright year errors in master

Summary
Details and comments

Yes, the squash/merge done when a PR is merged rebases your code and creates a brand new commit that now has year 2021. Then even the master CI fails.

Yes, the squash/merge done when a PR is merged rebases your code and creates a brand new commit that now has year 2021. Then even the master CI fails.


Update RELEASE  to upcoming release version

Changes
Set  RELEASE: ${{ inputs.release || 'v1.11' }} in actions/update-deps/action.yaml

/kind

Fixes #

Release Note

Docs

/lgtm
reminder to use this as new example for release docs


getting error in air-gap install on flannel step--- waiting for Flannel to become healthy Error from server (BadRequest): container "install-cni-plugin" in pod "kube-flannel-ds-z7p52" is waiting to start: PodInitializing An error occurred on line Would you like to provide a support bundle to aid us in avoiding similar errors in the future? (y/N)

how to get through this

HI @bakudih,
You might not using the latest version. However, please see that for we are able to help you out when we face problems we need to have further information to be able to reproduce the issue (as requested in the other issue open by you https://github.com/replicatedhq/kURL/issues/4096 )
Therefore, could you please provide:

Please, provide the kURL logs which are under /var/log/kurl. Could you please add them here?
Please, provide the link used to the install
Please, let us know if it is a new install or if it is an upgrade
Also, can you please share the support bundle? It probably was created when the error was faced if say Y I want report the issue. Or you can see https://community.replicated.com/t/kurl-how-can-i-generate-a-support-bundle-when-my-kubernetes-cluster-is-down/911 and attached here the result.

Hi @bakudih,
Since the required info has not been provided and it is open without interaction since Feb 15 I am closing this one. However, please feel free to raise new issues if you need.
Thank you for your attention and time.


Expandable section

Need an expandable container that allows content to be shown or hidden. Current design can be found here: http://www.patternfly.org/pattern-library/forms-and-controls/expand-collapse-section/#/design
Visuals to refine design approach. We should have all the CSS to create this.
This can be seen on a form but might have other use cases.

Expansion Behavior Visuals

What ended up happening to this issue? Looking for a status. Thanks! :)


Release v0.20.1

Failure is another test flicker.  I tend to think all the unsafeRunSync in blaze tests is the culprit here, but it's a big job to rip out.


Update README

Add the effective evaluation paper
Add links to the SIMILAR and effective eval papers
Remove the results right now from the main readme and rather link to the benchmark folders for each case. Each folder will have the detailed results for the corresponding case.

We also need to group sections of the readme to prevent an information dump on the front page. Perhaps we could move benchmarking results to the benchmark folder, among other things.


(JS-0116) async function should have await expression

Description
A function that does not contain any await expressions should not be async (except for some edge cases in TypeScript which are discussed below). Asynchronous functions in JavaScript behave differently than other functions in two important ways:
Occurrences
There are 5 occurrences of this issue in the repository.
See all occurrences on DeepSource → deepsource.io/gh/josephistired/DiscordBot-v14/issue/JS-0116/occurrences/

Fixed -
Upcoming v1.0.3 release


Images change to brown color in tour

Got a similar brown image, when I moved around/made it full screen it showed the proper pano.
Fixed it making viewer global and then calling the resize() function.
var viewer = pannellum.viewer('panorama', data_json);
viewer.resize();  //Call this when you navigate to new scene


[Bug] Users can create a new variant with an empty name

Describe the bug
Users can create a new variant with an empty name. However the created variant becomes invisible after reloading the page.
To Reproduce
Steps to reproduce the behavior:

Go to the Playground
Click on the new variant button
Select a variant
Click the OK button
The new variant with an empty name is created

Expected behavior
A new variant with an empty name shouldn't be allowed.
Screenshots

Desktop (please complete the following information):

OS: macOS Monterey
Browser Chrome
Version: 116.0.5845.179

If more information is needed, please let me know, and I'll provide it.

WIP

Opened a PR #578


Resolve alias in Jest not working

Issue Type

Bug (https://github.com/react-boilerplate/react-boilerplate/blob/master/.github/CONTRIBUTING.md#bug-reports)

Description
I'm having same problem described here https://github.com/react-boilerplate/react-boilerplate/issues/1580 with last version of the boilerplate.
I can't re-open/comment the previous issue so I'm creating a new one.
Steps to reproduce

create a  folder named config inside app with 2 files in it, development.js and test.js first one being just

module.exports = {
  projectName: 'development',
};

second being
module.exports = {
  projectName: 'test',
};

add the following alias inside resolve in webpack.base.babel.js pointing to the newly created file:

alias: {
      config: path.join(process.cwd(), `app/config/${process.env.NODE_ENV}.js`),
},

reference this alias, for example inside the FeaturePage by adding import config from 'config'; on top of the file and <H1>{config.projectName}</H1> within the render method.

run the project (npm start) and open feature page: text development text is shown within the H1 block.

run test (npm run test); all the tests will pass but one returning:

FAIL  app/containers/FeaturePage/tests/index.test.js
  ● Test suite failed to run

    Cannot find module 'config' from 'index.js'

      at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:210:17)
      at Object.<anonymous> (app/containers/FeaturePage/index.js:111:41)

Versions

React-Boilerplate: 3.6.0
Node/NPM: 5.6.0
Browser:

Thanks for filling out the issue template!
I ran your steps to reproduce and couldn't produce the same error. Can you confirm you're resolving it correctly?
I ask because of the err you're getting.
Cannot find module 'config' from 'index.js'

You should be importing config in index.test.js like this
import config from '../index'

If not that, double check all of your code because I couldn't reproduce it on my end.

I forked the repository, you can clone it from https://github.com/dariospadoni/react-boilerplate/ and npm run test to reproduce the issue

It looks fixed, I just add to update jest-cli to last version...

Interesting. Thanks for dropping this, I'll keep an eye out for any other related issues and see if an update to jest-cli is merited

Actually, I noticed that in my several attempts, I also added
"config$": "<rootDir>/app/config/test.js"

to moduleNameMapper within jest config in package.json.  I couldn't, but I definitely think it might be something.

If this can help replication, you can clone the repo https://github.com/dariospadoni/react-boilerplate/
which I forked. Running npm run test will cause the error, adding "config$": "<rootDir>/app/config/test.js" will fix it (using `"jest-cli": "23.1.0")

I won't have time to look into this in detail right now but could you try fixing it with this:
{
  alias: {
    path.resolve(__dirname, `app/config/${process.env.NODE_ENV}.js`),
  }
}

No it does not fix, "config$": "<rootDir>/app/config/test.js" is still needed

I have the same issue but in CI (Jekins) . It works locally. I have tried all sorts of combination for moduleNameMapper in jest package.json and config alias in webpack. It still says could not find module config.


Add option to add base URL for zenml server (with support for cloud)

Describe changes
I implemented changes to allow

people to set base URLs for their ZenML server
for ZenML Cloud tenants to set the same along with an option to add organization ID and tenant ID to it to construct a full  cloud URL.

This is exposed through the /info endpoint of the server. The use case is to allow ZenML to use the URL to construct addresses for stuff like pipeline runs and other info on the server.
Pre-requisites
Please ensure you have done the following:

[ ] I have read the CONTRIBUTING.md document.
[ ] If my change requires a change to docs, I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
[ ] If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Other (add details above)

I think for the future we will also need the org id separately, I would not make it part of the base url.
Base url should be: https://cloud.zenml.io/
then the tenant should also have its organization_id available to it to construct a full url
I believe the tenant_id is already there


Request validation fails when POSTing an array with only one element

/tags:
    x-swagger-router-controller: tags
    post:
      operationId: addTags
      parameters:
        - in: body
          name: tags
          required: true
          schema:
            type: array
            items:
              type: number
              format: int32
      responses:
        "200":
          description: Success
        default:
          $ref: '#/responses/ErrorResponse'

This simple definition works good for empty arrays and arrays with multiple items.
curl -H "Content-Type: application/json" -X POST -d '[]' http://localhost:8080/tags
curl -H "Content-Type: application/json" -X POST -d '[1,2]' http://localhost:8080/tags

But it fails for array with only one element.
curl -H "Content-Type: application/json" -X POST -d '[1]' http://localhost:8080/tags

{
  code: 'INVALID_TYPE',
  params: ['array', 'integer'],
  message: 'Expected type array but found type integer',
  path: []
}

As I understand it, the reason lies here:
  // Attempt to parse the string as JSON if the type is array or object
  if (['array', 'object'].indexOf(type) > -1) {
    try {
      value = JSON.parse(value);
    } catch (err) {
      // Nothing to do here, just fall through
    }
  }

JSON.parse([1]); returns integer 1
I also see that we have a handler for such situations. It will work If I add collectionFormat: multi. But as I see in http://swagger.io/specification/, we can use collectionFormat only "If in is any value other than body".

This has been fixed here: https://github.com/apigee-127/sway/pull/114  I'll be putting together a release this week that should fix this.

@whitlockjc Wow. Sounds good. Many thanks

Need #114 to be fixed (just reseting some conflicting changes) then we can merge that to fix this...

This was fixed using 94ba34fe8008e9febdff2ff1ee48027ac95a881f.


Start server but got 404 error

After model-loading, I got a 404 error. What happened?

hi, which demo you run? streamlit or gradio? I think the problem maybe you use GET method, and server starts with POST method. Use example: https://github.com/gpt-omni/mini-omni/blob/main/webui/omni_streamlit.py#L127

OmniChatServer code in the server.py, which use a post method for /chat api


build(protocol-definitions): Add ./lib/legacy export path

Adds the ./lib/legacy export path to protocol-defs, replacing alpha.

Superseded by other changes; closing.


Update list and create descriptions for the Components and Layout Patterns list on overview pages

Objective
Our current Components and Layout Patterns overview pages do not reflect what we are offering via the Storybook. We need to make updates to make sure the adopters know what we are offering so that they can adopt.
Task

Update the list and links on the Components and Layout Pattern overview pages.
Create the list description for the Components and Layout Pattern list items.
Review the update with Olivia and Lara.

Acceptance criteria

[x] The list and description are updated and live on the Cupcake website

FYI: @oliviaflory @jacobottesen-dgc

Task update: The list on both overview pages is updated and will work on the list description next.

Task update:

List update completed.
The list description work document can be viewed here.

@larahanlon2 The Layout Patterns list description is ready for view and feedback and can be found here.
@oliviaflory The Components list description is ready for view and feedback and can be found here.
Please feel free to make any edits and let me know if you have any questions.

FYI:

The list on both overview pages is composed based on the list in Storybook.
Our goal is to make the update with the finalized description on the website by next Tuesday.
The description will also be added to the storybook DOC tab and used as the Intro section copy when we are crafting content for its pages.

Status update: Description added to the website. Preview can be found inside of the PR 507.
The Layout Patterns list description can be viewed here.
The Components list description ican be viewed here.

FYI: @ljcarot

Description is live on the Cupcake website.

Thank you all for the hard work and last minute push to get these pages live!


353 absolute path fix

Closes donejs/donejs#353

Coverage decreased (-0.8%) to 96.653% when pulling 97475f247be951bf44269926b088caa1d6a9420d on 353-absolute-path-fix into c6cc995b4231f905e7e4b990499910e8f6beba27 on master.

Coverage increased (+0.03%) to 97.468% when pulling f83092ad5bba0ef3d5843a99e2cb81c51d7e20c5 on 353-absolute-path-fix into c6cc995b4231f905e7e4b990499910e8f6beba27 on master.

Coverage increased (+0.07%) to 97.51% when pulling 3358a8c3ba250de595c697cfe9acf7f636d59804 on 353-absolute-path-fix into c6cc995b4231f905e7e4b990499910e8f6beba27 on master.

Coverage increased (+0.07%) to 97.51% when pulling cc76bfac265f56e8615da607014f988af2fdf71c on 353-absolute-path-fix into c6cc995b4231f905e7e4b990499910e8f6beba27 on master.

Coverage increased (+0.07%) to 97.51% when pulling 394b90abddbe36a302b99afeca42e137fd1db1d1 on 353-absolute-path-fix into c6cc995b4231f905e7e4b990499910e8f6beba27 on master.

Coverage increased (+0.07%) to 97.51% when pulling 2a0aa2a983cf38f7128f351361b1204afb00f029 on 353-absolute-path-fix into c6cc995b4231f905e7e4b990499910e8f6beba27 on major.

Coverage increased (+0.07%) to 97.51% when pulling 9fca138805f0033686a3bfbe55c326c8116cf46d on 353-absolute-path-fix into c6cc995b4231f905e7e4b990499910e8f6beba27 on major.


add project avatar

Change Summary

Add project avatar to the menu bar [ resolves #114 ]

should work like this:

placeholder of kibibit if no project is open
project logo if a .logo.png file is included in the same path as package.json (will be supported once we parse some project details on the server)
show a folder icon if a folder is selected that doesn't have a package.json file + .logo.png file to read

[x] did you link this PR to an issue?
[x] did you lint your changes to both javascript and scss?
[x] "I'm pretty sure I'll be able to read and understand this PR, even if I wasn't the author." - said the PR author

This change is

LGTM

3 hound violations... :(
how do I unLGTM?

Lol I'll fix it. I just wanted to try something so I removed some things to check if hound stopped notifying on depth of selectors
You unLGTM WITH one of the following:
**reject PR**
**reject this PR**
cancel LGTM
 cancel my LGTM
 :lgtm_cancel:

cancel LGTM

reject PR

now there's no violations
tell me if you want me to fix anything else or give me a LGTM 👂

Very Nice
LGTM

LGTM again

LGTM !

thx Or :-)


fix hoisting

Current dependencies on/for this PR:

main

PR #14   👈

This stack of pull requests is managed by Graphite.


Add key missing metadata to LINZ STAC extension

Enabler
So that we can validate the key LINZ STAC properties, we want to add key LINZ metadata properties to the LINZ extension.
Acceptance Criteria

[ ] Following rules added to the extension
[ ] 'security_classification' property (Enumeration - see levels here for values) - add and make mandatory
[ ] Make the 'version' community extension mandatory for all datasets
[ ] Make the 'quality' extension mandatory for all datasets see #843

If these are not going to be adopted by teh community:

[ ] 'providers' property (role = 'manager') - copy schema rules from 'providers' core objects and make mandatory
[ ] 'lineage' property (Text) - add and make mandatory
[ ] 'lifecycle' property (Enumeration "under development", "preview", "on going", "completed", "deprecated") - add and make mandatory

Additional context
See new metadata analysis. We can probably combine our rules with the STAC JSON schema easily.
Tasks

[ ]
[ ] Add tests
[ ] Update README
[ ] Update JSON Schema

Definition of Ready

[ ] This story is ready to work on, according to the
team's definition

Definition of Done

[ ] This story is done, according to the
team's definition

FYI @palmerj @dwsilk

@l0b0 would you have time to edit the acceptance criteria for this one and add an estimate before planing this afternoon?

Needs to be estimated at refinement today.

We might've already discussed this, but I think the security_classification property should be part of a separate extension, since it isn't completely LINZ-specific. This issue would then be about making it mandatory.

In the discussion, which we recorded in the confluence page, we decided to leave it in the LINZ extension for now as we weren't sure anyone else would want to use it and to keep it relatively simple for now.
I don't think many (any?) of these things will be completely LINZ specific more just a collection of things that are useful for LINZ. Could we take this approach now and move it to a separate extension later? Or would that be problematic?


Implement string hash code in C.

SPARKR-135. Improve wordcount performance by implement the string hash code in C. The performance improvement is about 10X.
On a 3-worker-node Spark cluster, each node has 16 threads (2.93Ghz Xeon), 48G memory:
3.2 G data  from 96 minutes to 8 minutes
320 M data from 10 minutes to 1 minute
@shivaram I haven't test this patch on Windows, could you help to test it? Thanks.

This patch works well on my laptop. But when I compiled the pkg, for the first time, on a fresh installed server. An error occurred to me. That is:

** testing if installed package can be loaded
Error in dyn.load(file, DLLpath = DLLpath, ...) :
unable to load shared object '/home/todd/git/SparkR-pkg/lib/SparkR/libs/SparkR.so':
/home/todd/git/SparkR-pkg/lib/SparkR/libs/SparkR.so: undefined symbol: IS_SCALAR
Error: loading failed
Execution halted
ERROR: loading failed

The server is running ubuntu 14.04 32bit, with R version 3.0.2 (2013-09-25).

@sun-rui can probably give more details, but I think this is due to the older R version (3.0.2) not having the IS_SCALAR symbol. Could you try with a newer version say 3.1.2 ?
Also we should try to see if we can just symbols present in earlier R versions if this is the problem.

It works after I upgraded R to 3.1.2 !
By the way, is there anything like make clean? I had to rm *.so *.o by hand now.

@7c00, you can "cd pkg/src; make clean"
I was not aware that the IS_SCALAR macro is not supported by earlier versions. But I think this is not an urgent issue to make the code work with those versions. If we receive more user's feedback for using older versions, I can find a portable solution.

@sun-rui Nice! :clap:
I think it would be better to add some notes or suggestions on R version to README.md before a portable solution comes up.

The DESCRIPTION file lists the R version required by a package. Technically we only require R >= 3.0, and I think its good to be backwards compatible with a few R versions.
One simple fix could be to just import the definition to our file from https://github.com/wch/r-source/blob/70e7151b62fd95ab5660c040073d746073b11032/src/include/Rinternals.h#L319
Lets open a JIRA to track this.

Submitted https://sparkr.atlassian.net/browse/SPARKR-166 for this issue.


Feature/mongoid7

adapt to mongoid v7.0

the ci failed, but not on the unchanged old mongoid branches

@CI PLS REBUILD

There're a few references to versions of mongoid in README. Also please update https://github.com/mongoid/mongoid-locker/blob/master/CHANGELOG.md.

Documents updated:

README.md
CHANGELOG.md
LICENSE.txt (year part)

@dblock PLS re checking it.

Looks good, just that change thing pls.

Merged, thanks.

@dblock When release a new version?

Opened https://github.com/mongoid/mongoid-locker/issues/54, any maintainer can do, I might get to it soon


Bug is not triggered

I am trying to trigger the bug by the tools out of the box. But it looks like it is not easy to do. As I already mentioned in the previous report that the citadeld cannot be stopped on Android 11 (with October patches). The EC image was downgraded but not verified yet that the downgraded version is launched on boot up. I say "was downgraded" only in assumption that the 'fastboot oem citadel rescue command' returned 'OKAY' error code.
Now I am trying to to dump the Boot ROM. No luck so far. Any leak subcommand returns fixed 12 bytes value:
./nosclient leak 0x0 0x16
08 01 18 01 20 04 2a 04 00 00 00 00

To get more debug information I modified that frida script to intercept the nos_call_application() directly in the 'nosclient' tool.
 Bad EC image stored in the Titan M?

Now it is clear why the bug is not triggered. Indeed, the affected EC image is stored only in inactive slot. So, I have to downgrade the active slot as well.
 Nice to see that someone is using our tools :)
One problem you have is that the system automatically updated the firmware at next boot after you downgraded it. One solution could be to downgrade Android to a version embedding the same or lower version of the firmware. Another solution is to patch the system so that Android won't run citadel_updater at boot (you can patch init or rename the binary file).
A second issue you probably have is that citadeld is running. When playing with our client, we always had issues in the cases where citadeld is running. This is why we recommend to kill/stop the service.

Great! Glad to hear it is possible.

You mentioned "A second issue you probably have is that citadeld is running. When playing with our client, we always had issues in the cases where citadeld is running. This is why we recommend to kill/stop the service."
I tried to stop the citadeld by 'stop vendor.citadeld' but without success. Please, see the open ticket #2. To bypass this limitation, I use the debugger to hook the CheckControlPropertyPerms() function by replacing return code on any non-zero value.

I have replaced the ec.bin and ec.rec directly in the vendor partition.
blueline:/vendor/firmware/citadel # strings ec.bin | grep brick
brick_v0.0.8232-b1e3ea340
brick_v0.0.8232-b1e3ea340 2020-09-25 16:57:52 wfrichar
brick_v0.0.8232-b1e3ea340
brick_v0.0.8232-b1e3ea340 2020-09-25 16:57:52 wfrichar

Now, looks like RW_B has a suitable EC image to be flashed.
Chip:    Google Citadel C2-PVT
Board:   0
RO_A:    0.0.3/d55cc99c ok
RO_B:  * 0.0.3/874a9517 ok
RW_A:    0.0.3/brick_v0.0.8232-b1e3ea340 ok
RW_B:  * 0.0.3/brick_v0.0.8232-b1e3ea340 ok

But it still doesn't leak information:
blueline:/data/local/tmp # ./nosclient leak 0x0 0x16
08 01 18 01 20 04 2a 04 00 00 00 00

Cool! After renaming the init_citadel it stared working!
blueline:/data/local/tmp # ./nosclient leak 0x0 32
00 00 02 00 99 14 00 00 b9 3e 00 00 b9 3e 00 00 b9 3e 00 00 91 3e 00 00 b9 3e 00 00 b9 3e 00 00 b9 3e 00 00 b9 3e 00 00 b9 3e 00 00 b9 3e 00 00 b9 3e

Great! Enjoy :)

It is so exciting project!
I just recently thought about how to look at the runtime state of the Titan M. And here it is. It is real!
Thanks for your amazing work!

What are the read-memory access limitations and how they are managed? And what is the strategy of dumping the whole memory map?
I noticed that sometime the 'leak' command hangs. After resetting, the memory region being dumped gets accessible. Or it is exclusively the side effect of running the exploit?
For example:
blueline:/data/local/tmp # ./nosclient leak 0x2000 0x16
^C.       // no output, interrupt running command here after several seconds
130|blueline:/data/local/tmp # ./nosclient reset
blueline:/data/local/tmp # ./nosclient leak 0x2000 0x16
11 40 da 68 11 40 1a 69 11 40 5a 69 11 40 9a 69 11 40 4f 4a db 69 // immediately returns the memory content

We describe the memory regions we know about in the white paper, but the full memory mapping is still unknown. Some memory area may not be mapped or simply not accessible with read permissions (the leak command won't try to change the permissions). Also the exploit will sometimes make the chip hang. Resetting the chip can be necessary in that case.

More likely, access to the non-mapped address causes an exception in the Titan M. I guess they has 1 byte granularity of the memory as the SoC doesn't have the MMU.
White paper states that the BootRom is mapped to the 0x0 - 0x3fff.
Actually, the last accessible byte is mapped at the 0x3fdf. Higher memory address are not available until RO_A
blueline:/data/local/tmp # ./nosclient leak 0x3fdf 1                           // last byte of the BootRom mapped to the Titan M
ab
blueline:/data/local/tmp # ./nosclient leak 0x3fe0 1                          // empty return. Nothing is mapped here. Probably causes an exception.

130|blueline:/data/local/tmp # ./nosclient leak 0x3ffff 1                   // just behind the RO_A. Also non-mapped

blueline:/data/local/tmp # ./nosclient leak 0x40000 1                       // First byte of mapped RO_A.
fe

Yes the BROM is not exactly ending at 0x3fff. We used this address in a schema for the presentation. The actual end is indeed 0x3fdf.


Fix #185

Yep, the bug here is caused by the wiring preprocessor basically renaming .ino to .cpp. Maybe it could also provide a mapping file telling which files were renamed to what, so the later stages could use it instead of guessing


Goobi S3 storage: Adjust OCR taskmanager plugin

We are not sure if this is still needed, as most OCR is created by the Internet Archive. @jtweed could you find out / let us know if you still need OCR done by the intranda service. If not, this one can be closed.

Hi @opaetzel. I've done a bit of digging on this and it looks like we can certainly de-prioritise this. We'd like the option of being able to use the OCR service in the future, for archives, but it's not something that needs to happen before the migration to AWS.

not needed anymore


user and group name functionality

This is more of a feature request than a bug. I'd like the ability to change the group and user of a file that will be packaged into the RPM.
Currently you can change the fileMode but I cant see how to set the group or the user.
Thanks
S

You should be able to do this:
user = 'foo'
group = 'bar'

I've done this personally at the root of the task, but this should also work at the filespec level.
This should probably be added to the README.

I note that this may collide with the 'group' at the RPM level.  Hrm...


Bug 1830031: Update OLM to use UID for OG Labels

Problem:
OLM is using an older version of the operator-framework/api repository
with which OLM applies a label with a blank value whose key is equal to
"olm.operatorgroup/.".
Kubernetes limits the lengths of label values and keys to 63 characters.
This limit can easily be overcome when the OperatorGroup has a long name
or when it is deployed in a namespace with a long name.
Solution:
Update OLM to use import the latest version of the operator-framework/api
repository that uses "olm.operatorgroup.uid/" as the key
for OperatorGroup labels. The length of this label will always be 58
characters as UIDs are 36 characters long.

/approve

It looks like a libcontainer package using CGO is being transitively pulled in via the latest operator-registry: https://github.com/operator-framework/operator-lifecycle-manager/pull/1496/checks?check_run_id=637003810, which breaks the upstream image build. It has +nocgo implementations, though.

Looks like there's a legitimate E2E failure -- OpenShift Prow was broken over most of the weekend and incorrectly marking failing checks green.
/test e2e-aws-olm

/hold

/hold cancel

/retest

/lgtm

/hold

/unhold

/test unit

/retest

/retest

/lgtm

/approve

/cherry-pick release-4.4


Development page out of date + possible additions

The "Development & Contributing" page at https://docs.teslamate.org/docs/development/ still refers to Postgres 10, whereas the main installation page suggests v12 is required.
Additionally it doesn't note a requirement for Grafana, or node.js which are both listed on the install page.
As someone currently working on a PR - it would be helpful if the contributing page covered:

How to test migrations (mix ecto.migrate ?)
How to refresh grafana dashboards from updated definitions (just re-run/grafana/dashboards.sh restore?)

Thanks!

I must admit I agree with Lee.  When I first started building a Dev environment and writing code I ended up googling most of the software/dependencies and commands.  A lot of trial and error ensued.
I develop on Windows and it took me the best part of a month to build a Windows VM which was scripted and could be consistently rebuilt from nothing to a consistent DEV environment.  All the commands on the help page are for Linux and some don't work on Windows, such as  /grafana/dashboards so I ended up having to write my own scripts to replace it.
Overall the gate is set very high with regarding learning and building a Dev environment and it probably puts a lot of people off.
A good example of missing instructions is having to build test scripts.  The developer page states you have to run "mix test" before submitting, however it doesn't say that you may need to add extra tests to your test script to cover your code changes.
Ideally it would be good to have a tutorial of how to build for Linux and Windows and then tutorial taking you through the steps of development with all the relevant commands you will need.

Just to follow up on this. I managed to get everything setup I think, but I still had some issues, particularly around tests.
It seems like whenever I run mix test it insists on doing it against the main database, not the test DB. Setting MIX_ENV didn't seem to affect that, so clearly I'm doing something wrong, and I'm not sure what.
I've just noticed that the website looks like it's in the repo, so when I get chance I'll put together a PR to reorganise some of the content, and hopefully we can evolve it from there!

I'm in the initial phase of setting up a development environment.
I'm actually already stuck at deciding whether to use Windows or Linux :)
It seems on the previous comments that Linux would be the way to go. Would WSL2 work?
Does anyone have their own guide they want to share?

I've just noticed that the website looks like it's in the repo, so when I get chance I'll put together a PR to reorganise some of the content, and hopefully we can evolve it from there!

Still waiting for your PR :-)

It's unlikely to appear - sorry - I don't think I have the knowledge to add the missing information.

It's unlikely to appear - sorry - I don't think I have the knowledge to add the missing information.

OK, I updated the requirements in #4017
Testing with our CI is already included since #3856


Use proxies and User agent for kartaview downloader

I get the following error after downloading a few hundred images:

Error: HTTPSConnectionPool(host='storage13.openstreetcam.org', port=443): Read timed out

Using proxies and user agents might solve this issue.

KVDownloader already has user agents and proxies. It might be due to other reasons.
def worker(row, output_dir, cropped):
            url, panoid = row.url, row.id
            user_agent = random.choice(self.user_agents)
            proxy = random.choice(self.proxies)

            image_name = f"{panoid}.png"  # Use id for file name
            image_path = output_dir / image_name
            try:
                response = requests.get(
                    url, headers=user_agent, proxies=proxy, timeout=10
                )
                if response.status_code == 200:
                    with open(image_path, "wb") as f:
                        f.write(response.content)

                    if cropped:
                        img = Image.open(image_path)
                        w, h = img.size
                        img_cropped = img.crop((0, 0, w, h // 2))
                        img_cropped.save(image_path)

                else:
                    if self.logger is not None:
                        self.logger.log_failed_pids(panoid)
            except Exception as e:
                if self.logger is not None:
                    self.logger.log_failed_pids(panoid)
                print(f"Error: {e}")

Seems like this problem is caused by the issue mentioned in #98. So I'll close this now.


Add usage documentation to README.md.

Pull Request Test Coverage Report for Build 392

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 97.743%

Totals

Change from base Build 389:
0.1%

Covered Lines:
1689

Relevant Lines:
1728

💛  - Coveralls

Pull Request Test Coverage Report for Build 392

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 97.743%

Totals

Change from base Build 389:
0.1%

Covered Lines:
1689

Relevant Lines:
1728

💛  - Coveralls


Perfomance

It occurs on startup and during all the time the editor is opened. Maybe it's because of some extension. I do not know exactly.

As I wrote on another issue related to yours:

I'm experiencing similar issues. Opening VSCode after minimize feels so sluggish and often goes laggy when I write code. When I ALT+TAB between applications and VSCode, I'm very often greeted with a black screen that takes 1-2 seconds before disappearing.

My problem was related to json-server.

@vtivadze Can you give some more details and steps to reproduce? Did it happen when you opened a specific file, using a certain JSON schema?

It happened when I simply opened the VSCode.
I've just renamed the JSON service folder and the problem disappeared.
On Tue, Jul 12, 2022 at 1:59 PM Martin Aeschlimann @.***>
wrote:

@vtivadze https://github.com/vtivadze Can you give some more details
and steps to reproduce? Did it happen when you opened a specific file,
using a certain JSON schema?
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode/issues/154710#issuecomment-1181564940,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACQAL2HDITO7E7J7YJAT6X3VTU6YRANCNFSM53FJYC2Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Vakhtang Tivadze
+995 568 816 236

What do you mean by the JSON service folder?

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!

Somewhere in the VSCode folder structure there was a folder for JSON
service or server.
I do not remember exactly.
On Wed, Jul 13, 2022 at 5:13 PM Martin Aeschlimann @.***>
wrote:

What do you mean by the JSON service folder?
—
Reply to this email directly, view it on GitHub
https://github.com/microsoft/vscode/issues/154710#issuecomment-1183207449,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACQAL2C2LXHKZPCA67JSPDTVT26GHANCNFSM53FJYC2Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Vakhtang Tivadze
+995 568 816 236


simple github authentication - record not unique

I'm working on authentication using Devise, trying out some of the strategies.
Following the tutorial at https://www.digitalocean.com/community/tutorials/how-to-configure-devise-and-omniauth-for-your-rails-application
I added github github authentication, having created an app in my github account.
In the app, we get the following screen which greets us :

The auth bit works (http://localhost:3000/users/auth/github), but the callback is not working. We get the problem :
ActiveRecord::RecordNotUnique in CallbacksController#github
The callbackscontroller is
class CallbacksController < Devise::OmniauthCallbacksController
    def github
        @user = User.from_omniauth(request.env["omniauth.auth"])
        sign_in_and_redirect @user
    end
end

ok, it seems like the problem is that the user is not being created when login happens :
_csrf_token: "Y/DRWtS7zjGYSVFjIiAoA1tjjrOXiyEn9YL7mK8p/e4="
flash: {"discard"=>[], "flashes"=>{"alert"=>"You need to sign in or sign up before continuing."}}
session_id: "8bad276d2472f7fbb82906283004bce6"
user_return_to: "/"
warden.user.user.key: [nil, "$2a$11$wS/FLrPTI.hKbP26rH2M0O"]

This seems obvious now - we need to have  a sign up section, or authorise everyone automatically.


feat(cli): add "--sort" option for branch and tag to support sorting

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

9 of 15 (60.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 87.073%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

tensorbay/cli/cli.py
4
6
66.67%

tensorbay/cli/utility.py
5
9
55.56%

Totals

Change from base Build<PHONE_NUMBER>:
-0.03%

Covered Lines:
7517

Relevant Lines:
8633

💛  - Coveralls


About un-uploaded files

I missed to upload some files about tcp_socket class. I'll upload them ASAP. Sry for my mistake

keilo/source/socket/tcp_socket.cpp
keilo/source/socket/tcp_socket.hpp
Done


keras-mobilenet-yolov3

I've trained yolov3 on mobilenet backend and get a good MAP(about 90M and 2 times faster than keras yolov3).I upload my code for mobilenet-yolov3 on https://github.com/Adamdad/keras-YOLOv3-mobilenet.git
use it as you wish

咱都是国人哈哈哈哈为啥用英文哈哈哈哈


Board Reports: Filter by Fiscal Year not working

On the Board Reports page, "Filter by Fiscal Year" is returning reports which are not from the selected fiscal year.

The filters are working as expected. Here's what's going on:
(1) Each bill receives a "legislative session," based on the MatterIntroDate, given in the Legistar API.
For example, Board Report 2016-0499 has a MatterIntroDate of "2016-06-06": http://webapi.legistar.com/v1/metro/matters/3016. Thus, this board report occurs in the 7/1/2015 to 6/30/2016 fiscal year (i.e., legislative session).
(2) The date displayed in the search results represents the most recent action item. Again, with Board Report 2016-0499, the most recent action date was in April of 2017.
I think we can do a few things to clarify the search results:

add explanatory text next to the date saying something, like, "most recent action"
add a line for "Fiscal year" below or above the last action date - it may look redundant when adding a filter, but I think it could clarify the results.
show the "Matter Intro Date" in the results and on the bill detail (N.B. This fix would require making updates to the scraper and our data import functionality – so, it's a much more involved change.)

@shrayshray - what would you prefer?

@reginafcompton let's go with the first option:
•	add explanatory text next to the Board Report date saying "most recent action" similar to mock-up below.

@shrayshray - explanatory text on staging site! https://lametro.datamade.us/search/

@reginafcompton good to go!


Generate SLSA provenance for builds

With https://github.com/slsa-framework/slsa-github-generator we should be able to build SLSA-compliant provenance. I can then look into what else we need to do to reach SLSA Level 3.

Hey @craigbox I have some with generating SLSA Provenance, I would like to work on this 🙂

Thanks @lakshya8066.  Why not write up a design document/enhancement proposal?

Hey, @craigbox! I have finished preparing a design document for this issue. Please have a look over it here.


What is metadata and how to use it

When looking at the code I can see there is a concept of metadata. That is, it's possible to add extra fields which aren't part of the standard defined set of fields.
I'm wondering if it's possible to add custom fields to the log records which are sent to logstash. If so, how do I do this? In the main wildfly/jboss configuration file?
I'm currently working with a lot of docker containers, each running a wildfly instance with an application. Instead of filtering on application based on the host address, I'd like to pass an extra field with the name of the application. Is this a proper use case for the metadata functionality or am I getting it wrong?
Apologies for using the issue tracker for help, but it seemed like the best option.

Yeah I think the metadata is the right use case. How were you planning on passing in the application name? Like would it be hard-coded in the metadata?
This is the right place for this conversation at this point. Once I work the issues out and it seems stable I'll bring this stuff into the log manager.

Alright... Yes, I was planning on hardcoding it in the Dockerfile and make sure it propagates to wildfly.
Where should it end up? What actually is the metadata?

The metadata is what ever you want it to be really. It's essentially just a map to add extra data to the formatter.
The syntax looks something like appName=app1.war,otherProp=otherValue. The formatter itself does allow the expression syntax of ${system.property.key:default-value} however there's a catch. Since both WildFly and formatter use the same syntax it will be resolved in WildFly and not the formatter.
That can be an issue if you change the system property frequently. What happens is a logging.properties file is written out with the hard-coded system property from the last boot. So on the next boot it will use the previous value until the logging subsystem kicks in. Hopefully that makes sense.
Anyway if you want to use a system property you can definitely do that.
/subsystem=logging/custom-formatter=logstash:write-attribute(name=properties, value={metaData="appName=${app.name:UNKNOWN}"})

Please let me know if you need further clarification. I'm not sure how well I'm actually explaining it :)

Thanks a lot! This is exactly what I needed...


Interface with tester and almost done InternalStepRunner

@bratseth another round.
Must be merged with:

https://git.ouroath.com/vespa/hosted/pull/6146
https://git.ouroath.com/vespa/vespa-yahoo/pull/1556

Well, the changes in vespa-yahoo mustn't  really be merged together with the others, because this isn't it use yet, but they do belong together :)


Detect rbd map timeout errors and detect if image is mapped, to undo the mapping

It is noted that there are cases when,

rbd map invocation times out #507
It leaves behind a mapped image to a device (because mapping actually completed, but state/success of the same was missed out)

This causes future attempts to map the image on other nodes, or attempts to delete the image, to error out stating image is in use.
This issue is being opened to add some safety checks and workarounds in the code, that will detect such timeout errors, and check to see if the image is mapped to a device, and undo the same.
NOTE: as we do not know the device path when rbd map returns a timeout failure, it may need the current PR #510 work to complete to leverage the detection of device using rbd device list instead of current schemes.

I think we can call unmap with image name also, why to wait for #510?

Yes, "rbd unmap" supports unmapping by image name.  No need to list mapped images just for that.

Ack! was not aware of that. We should use imagespec instead of hunting for the devicePath. Also, I assume this is true when --device-type is nbd?

Ack! was not aware of that. We should use imagespec instead of hunting for the devicePath. Also, I assume this is true when --device-type is nbd?

Yes, with the caveat for both where it doesn't handle different clusters.

@ShyamsundarR Yes, it should be true.
I'm not sure I understand the need to unmap the device in this case.  If "rbd map" times out because udev decided to take a break but the device is there (i.e. you listed mapped images and got the device node), would pretending that nothing happened not work?
If we keep encountering issues with udev, we could probably adopt this course of action for the general case as well -- make "rbd map" wait for udev for a much shorter period of time and then instead of immediately bailing with a timeout error, try to get the device node by listing the devices.

Yes, with the caveat for both where it doesn't handle different clusters.

This is another thing that I don't fully understand.  I thought we controlled image names in CSI?  In https://github.com/ceph/ceph-csi/pull/510#issuecomment-516074646 @ShyamsundarR mentioned that we indeed do use UUIDs there, but the discussion continued with various workarounds...

@ShyamsundarR Yes, it should be true.
I'm not sure I understand the need to unmap the device in this case. If "rbd map" times out because udev decided to take a break but the device is there (i.e. you listed mapped images and got the device node), would pretending that nothing happened not work?

So the manner in which kubernetes (and possibly other container orchestrators) behaves (and also laid out in the spec around timeout handling of various RPCs), is as follows,

Call NodeStageVolume with a timeout of t1
If call timed out, wait for some exponential backoff time t2
Call NodeStageVolume again
Bail out at some point in time from calling NodeStageVolume, because the workload moved to a different node, and kubernetes never got a success back for any of its calls (there is a race window here, irrespective of what we fix, in kubernetes, but we will pretend for the moment that it is not so)

As, a result never call NodeUnstageVolume for the same image (which would result in a leak of the image map)

In our case, if the first call "hangs" due to various reasons, its success/failure result from the RPC response maybe ignored if t1 is past. Hence leaving behind a mapped rbd device is not a desired resolution. It all depends on the timeout, which the plugin is unaware of (IOW, there are no call cancels, so the plugin will not know the RPC was cancelled/timedout).

If we keep encountering issues with udev, we could probably adopt this course of action for the general case as well -- make "rbd map" wait for udev for a much shorter period of time and then instead of immediately bailing with a timeout error, try to get the device node by listing the devices.

We could, but I am not comfortable deciding the timeout values to closely resemble or be lower than the kubernetes/CO timeouts and hence make this work within such slices.
Beyond above discussion, we could/should still add a timeout to detect udev (or other) hangs/failures faster (or use this to "fail fast") and hence prevent 2-10 minute hangs that we have seen.

Yes, with the caveat for both where it doesn't handle different clusters.

This is another thing that I don't fully understand. I thought we controlled image names in CSI? In #510 (comment) @ShyamsundarR mentioned that we indeed do use UUIDs there, but the discussion continued with various workarounds...

So the caveat here is that IF there are 2 images with the same poolname and imagename, but from different ceph clusters, imagespec based commands are inadequate to identify which image we are referring to.
The workaround or statement to say this will not happen, is that we use UUIDs to name the image, and this collision across cluster hence needs a UUID collision to occur as well, and that is mathematically very rare etc.

(I might be talking nonsense here because I'm not familiar with CSI RPCs, please bear with me.)

In our case, if the first call "hangs" due to various reasons, its success/failure result from the RPC response maybe ignored if t1 is past. Hence leaving behind a mapped rbd device is not a desired resolution. It all depends on the timeout, which the plugin is unaware of (IOW, there are no call cancels, so the plugin will not know the RPC was cancelled/timedout).

This makes me slightly concerned about the reliability of the system, but the fact that these RPCs must be idempotent is even more reason for UUID-based image names.  If the CO doesn't always cancel pending operations, stale mappings are unavoidable.
I guess I don't see why leaving behind a mapped device in this case is undesirable.  If "rbd map" times out but the device node is there, you send back a success.  If your reply is ignored and you get a retry, you can satisfy it immediately because the image with that (globally unique) name is already mapped.  If your reply is ignored and you don't get a retry (e.g. because the workload was moved to a different node by the CO), you get a stale mapping.  But it sounds like stale mappings/images/etc is something that the plugin should be prepared to deal with anyway because they are pretty much written into the spec...

Beyond above discussion, we could/should still add a timeout to detect udev (or other) hangs/failures faster (or use this to "fail fast") and hence prevent 2-10 minute hangs that we have seen.

Unfortunately, the 2 minute timeout in "rbd map" is not on the overall interaction with udev, but on the individual poll() call.  So in a scenario where multiple images are getting mapped concurrently and udev is really slow for whatever reason, "rbd map" could theoretically sit there for hours -- as long as udev keeps delivering events at a rate of ~0.5/minute.
What version of ceph were you testing with?  For 10 minute hangs, was that on 14.2.2 or something prior?

I have a dumb question, do CSI has to bother about the timeout error and handle it specially. it should be the client work to unmap the image if timeouts right?
@idryomov we can add some default timeout for rbd map command I will check whats the timeout for RPC call from Kube side. this should fix the timeout issue I believe
@idryomov any preferred timeout option from rbd side. as we don't know the image size, I think larger the image size it takes more time to map right?
@humblec ^^

@idryomov we can add some default timeout for rbd map command I will check whats the timeout for RPC call from Kube side. this should fix the timeout issue I believe

As @ShyamsundarR said, we probably don't want to tune our timeout (be it at a higher level in the plugin or at a lower level in "rbd map"/"rbd unmap") to various kubernetes timeouts because this is not reliable, both in general and (especially) at scale.

@idryomov any preferred timeout option from rbd side. as we don't know the image size, I think larger the image size it takes more time to map right?

No preferences, I would pick one based on test results on real hardware.
Currently, with just layering supported, the size of the image is completely irrelevant.  In the future, when we start supporting object-map and fast-diff, it might be a little more noticeable because the size of the object map is proportional to the size of image, but it is certainly not something that should be taken into account.  The timeout should be the same in all cases.

is there a way to handle this as this action belongs to rbd map and udev to map the image on a node. as  @idryomov  said earlier what if udev takes a break and maps the rbd device on the node, this will become stale device on the node which we cant delete (need manual cleanup)

I have a dumb question, do CSI has to bother about the timeout error and handle it specially. it should be the client work to unmap the image if timeouts right?

@idryomov your thoughts on this one?

I was deferring to @ShyamsundarR an @humblec because I'm not very familiar with CSI.  As I said in https://github.com/ceph/ceph-csi/issues/520#issuecomment-518263480, given the CSI spec, I think stale mappings/image/etc are unavoidable.
What is the client you are thinking of here?  The CO, the end user who just wants her PV, someone else?  If my reading of the spec is correct, only the plugin can be responsible for unmapping a stale mapping or removing a stale image because only the plugin has that information.  The CO may not have enough information to issue an "undo" RPC and on top of that the spec doesn't seem to require the CO to do that in all cases.

@idryomov as you said above it purely depends on udev to map the rbd image on the node ( as you said it can take a break and map the image later also) do you think this need to be handled in the code?
if yes I would like to check how this can be handled whether do we need to start a goroutine periodically and check the device is stale or not, if yes mark it for unmapping.
@humblec @ShyamsundarR Thoughts?

@Madhu-1 Are you referring to https://github.com/ceph/ceph-csi/issues/520#issuecomment-518263480 where I described a theoretical scenario for "rbd map" and udev?
A periodic check for stale images/mappings/etc sounds good, but it would be needed primarily to guard against the CO silently giving up and moving the workload to a different node, which is a separate issue.  The case of a CLI command taking forever (it could be any command that talks to RADOS, whether directly or indirectly through the kernel) is also separate -- it would likely take a per-invocation timeout in the plugin, but it is not clear to be that it is actually needed.  I'm generally wary of introducing arbitrary timeouts.
This particular ticket is about the unfortunate failure mode of "rbd map".  This timeout was slapped on as a poor workaround for a bug which has (hopefully) been fixed, so I think we should consider reverting it.  In any case, no need for an async goroutine here -- just look at the exit code or parse the error message, if it is a timeout, list the mappings and "rbd unmap" if needed.

In any case, no need for an async goroutine here -- just look at the exit code or parse the error message, if it is a timeout, list the mappings and "rbd unmap" if needed.

The above is what I would suggest as well, or maybe even checking if the device is mapped or not is optional, and we can issue a blanket unmap for the image-spec.

@Madhu-1 Are you referring to #520 (comment) where I described a theoretical scenario for "rbd map" and udev?
A periodic check for stale images/mappings/etc sounds good, but it would be needed primarily to guard against the CO silently giving up and moving the workload to a different node, which is a separate issue. The case of a CLI command taking forever (it could be any command that talks to RADOS, whether directly or indirectly through the kernel) is also separate -- it would likely take a per-invocation timeout in the plugin, but it is not clear to be that it is actually needed. I'm generally wary of introducing arbitrary timeouts.
This particular ticket is about the unfortunate failure mode of "rbd map". This timeout was slapped on as a poor workaround for a bug which has (hopefully) been fixed, so I think we should consider reverting it. In any case, no need for an async goroutine here -- just look at the exit code or parse the error message, if it is a timeout, list the mappings and "rbd unmap" if needed.

the image will be mapped immediately  if rbd map timeouts or it may take some time to map the image ( want to understand all the possible scenarios)

If "rbd map" fails because of this udev timeout, the image is definitely mapped -- it doesn't wait for udev events unless the mapping succeeds.  However, I think the kernel can return ETIMEDOUT as well and in this case the image is definitely not mapped.  You can distinguish between these cases by parsing the error message: a mapping failure will always have "rbd: sysfs write failed" in it, any other failure (e.g. udev timeout) won't.
This is why I suggested listing the mappings.  You don't need to parse "rbd map" output (thus relying on it to stay the same) and if you see the mapping, you expect success from "rbd unmap".  If you "rbd unmap" unconditionally, you don't know what to expect: success or "not a mapped image" failure.

Fixed in #561


Files() Supports sftp/ftp, and csv format

Feature request
files() Supports sftp/ftp, and csv format

@melin CSV format is in our plan, it will be released in V3.3 if possible.
About sftp/ftp, I want to know something more. Would you mind to share with me?

How about the data size? Can you use Stream Load to ingest the data?
What's the frequency to load the data? one-time for all data or daily loading?
How about the performance to get data from the ftp server?


Issue INSERT AN IMAGE not working

Have uploaded files in a folder on server of my provider.
Added a Basepath.
everything is working like a charm except INSERT AN IMAGE and INSERT A BACKGROUND IMAGE.
Does anyone have had the same issue and can give a hint how to fix it?

I experienced the same thing, but found out the insert image buttons started working when changing the Jquery version from 3 to 2 (or 1 should also be fine, as in the examples on the webpage). Didn't investigate it any further, though, so cannot tell what's wrong when using v3.